Post on 30-Dec-2015
Neural Networks Neural Networks ArchitectureArchitecture
Baktash BabadiBaktash Babadi
IPM, SCSIPM, SCS
Fall 2004Fall 2004
The Neuron ModelThe Neuron Model
Architectures (1)Architectures (1) Feed Forward NetworksFeed Forward Networks
The neurons are arranged in The neurons are arranged in separate layersseparate layers
There is no connection between There is no connection between the neurons in the same layerthe neurons in the same layer
The neurons in one layer receive The neurons in one layer receive inputs from the previous layerinputs from the previous layer
The neurons in one layer delivers The neurons in one layer delivers its output to the next layerits output to the next layer
The connections are unidirectionalThe connections are unidirectional (Hierarchical)(Hierarchical)
Architectures (2)Architectures (2)
Recurrent NetworksRecurrent Networks Some connections are Some connections are
present from a layer to present from a layer to the previous layersthe previous layers
Architectures (3)Architectures (3)
Associative networksAssociative networks There is no hierarchical arrangementThere is no hierarchical arrangement The connections can be bidirectionalThe connections can be bidirectional
Why Feed Forward?Why Feed Forward?
Why Recurrent/Associative?Why Recurrent/Associative?
An Example of Associative An Example of Associative Networks: Hopfield NetworkNetworks: Hopfield Network
John Hopfield (1982)John Hopfield (1982) Associative Memory via artificial neural Associative Memory via artificial neural
networksnetworks Solution for optimization problemsSolution for optimization problems Statistical mechanicsStatistical mechanics
Neurons in Hopfield NetworkNeurons in Hopfield Network
The neurons are binary unitsThe neurons are binary units They are either active (1) or passiveThey are either active (1) or passive Alternatively + or –Alternatively + or –
The network contains The network contains NN neurons neurons
The state of the network is described as a The state of the network is described as a vector of 0s and 1s:vector of 0s and 1s:
)1,0,0,...,1,0,1,0(),...,,( 21 NuuuU
The architecture of Hopfield The architecture of Hopfield NetworkNetwork
The network is fully interconnectedThe network is fully interconnected All the neurons are connected to each otherAll the neurons are connected to each other The connections are bidirectional and symmetricThe connections are bidirectional and symmetric
The setting of weights depends on the The setting of weights depends on the applicationapplication
ijji WW ,,
Updating the Hopfield NetworkUpdating the Hopfield Network
The state of the network changes at each time The state of the network changes at each time step. There are four updating modes:step. There are four updating modes: Serial – Random: Serial – Random:
The state of a randomly chosen single neuron will be The state of a randomly chosen single neuron will be updated at each time stepupdated at each time step
Serial-Sequential :Serial-Sequential :The state of a single neuron will be updated at each time The state of a single neuron will be updated at each time step, in a fixed sequencestep, in a fixed sequence
Parallel-Synchronous:Parallel-Synchronous:All the neurons will be updated at each time step All the neurons will be updated at each time step synchronouslysynchronously
Parallel Asynchronous:Parallel Asynchronous:The neurons that are not in refractoriness will be updated at The neurons that are not in refractoriness will be updated at the same timethe same time
The updating Rule (1):The updating Rule (1):
Here we assume that updating is serial-RandomHere we assume that updating is serial-Random
Updating will be continued until a stable state is Updating will be continued until a stable state is reached.reached. Each neuron receives a weighted sum of the inputs Each neuron receives a weighted sum of the inputs
from other neurons:from other neurons:
If the input is positive the state of the neuron will If the input is positive the state of the neuron will be 1, otherwise 0:be 1, otherwise 0:
jh
0 if 0
0 if 1
j
j
j h
hu
ij
N
jii
ij wuh ,1
.
The updating rule (2)The updating rule (2)
Convergence of the Hopfield Convergence of the Hopfield Network (1)Network (1)
Does the network eventually reach a stable Does the network eventually reach a stable state (convergence)?state (convergence)?
To evaluate this a ‘energy’ value will be To evaluate this a ‘energy’ value will be associated to the network: associated to the network:
The system will be converged if the energy is The system will be converged if the energy is minimizedminimized
j
N
jii
jiij uuwE1
, 2
1
Convergence of the Hopfield Convergence of the Hopfield Network (2)Network (2)
Why energy?Why energy? An analogy with spin-glass models of Ferro- An analogy with spin-glass models of Ferro-
magnetism (Ising model):magnetism (Ising model):
The system is stable if the energy is minimized The system is stable if the energy is minimized
01
00
000 0
00
0
1 1111
1 1 1 1
111
1 1
111
1 1 1
j
N
jii
jiij
jj
jjj
N
jii
iijj
j
jiji
ji
uuwE
eE
juhe
juwh
ju
dd
kw
1,
1,
,,
2,
2
1
system theofenergy potential overal The:
unit ofenergy potential The: 2
1
unit upon the exerted field local the:
unit ofspin the:
distance,
Convergence of the Hopfield Convergence of the Hopfield Network (3)Network (3)
Why convergence?Why convergence?ij
N
jii
ij wuh ,1
.
0 if 0
0 if 1
j
j
j h
hu
j
N
jii
jiij uuwE1
, 2
1
j
N
jii
iijj uwu1
, 2
1 j
jjhu 2
1
0 changenot will then 1 and 0 if jjjjjj hhuuuh
0 change will then 0 and 0 if jjjjj huuuh
0 changenot will then 1 and 0 if jjjjj huuuh
0 change will then 1 and 0 if jjjjjj hhuuuh
changenot do values if minimum is 2
1-E
changenot does when maximum is caseeach in
jj
jj
jjj
uhu
uhu
Convergence of the Hopfield Convergence of the Hopfield Network (4)Network (4)
The changes of E with updating:The changes of E with updating:
kkkoldknewkkoldkkj
jjknewkkj
jjoldnew huhuuhuhuhuhuEEE .2
1)(
2
1)
2
1
2
1()
2
1
2
1(
ij
N
jii
ij wuh ,1
.
0 if 0
0 if 1
j
j
j h
hu
j
N
jii
jiij uuwE1
, 2
1j
jjhu 2
1
0.2
1100 and 0 if
0.2
1100 and 0 if
0.2
1100 and 1 if
0.2
1010 and 1 if
kkknewkkoldk
kkknewkkoldk
kkknewkkoldk
kkknewkkoldk
huuuhu
huuuhu
huuuhu
huuuhu
In each case the energy will decrease or remains constant thus the system tends toStabilize.
The Energy Function:The Energy Function:
The energy function is similar to a The energy function is similar to a multidimensional (N) terrainmultidimensional (N) terrain
Global Minimum
Local MinimumLocal Minimum
Hopfield network as a model for Hopfield network as a model for associative memoryassociative memory
Associative memoryAssociative memory Associates different features with eacotherAssociates different features with eacother
Karen Karen greengreen
George George redred
Paul Paul blueblue
Recall with partial cuesRecall with partial cues
Neural Network Model of Neural Network Model of associative memoryassociative memory
Neurons are arranged like a grid:Neurons are arranged like a grid:
Setting the weights Setting the weights
Each pattern can be denoted by a vector of Each pattern can be denoted by a vector of -1s or 1s:-1s or 1s:
If the number of patterns is m then: If the number of patterns is m then:
Hebbian Learning:Hebbian Learning: The neurons that fire together , wire togetherThe neurons that fire together , wire together
),...,,()1,1,1,...,1,1,1,1( 321pN
pppp ssssS
m
p
jpp
iji ssw1
,
Limitations of Hofield associative Limitations of Hofield associative memorymemory
1) The evoked pattern is sometimes not 1) The evoked pattern is sometimes not necessarily the most similar pattern to the necessarily the most similar pattern to the inputinput
2) Some patterns will be recall more than 2) Some patterns will be recall more than othersothers
3) Spurious states: non-original patterns3) Spurious states: non-original patterns
Capacity: Capacity: 0.15 N0.15 N
Hopfield network and the brain (1):Hopfield network and the brain (1):
In the real neuron, synapses are distributed In the real neuron, synapses are distributed along the dendritic tree and their distance along the dendritic tree and their distance change the synaptic weightchange the synaptic weight
In hopfield network there is no dendritic In hopfield network there is no dendritic geometrygeometry
If they are distributed uniformly, the geometry is If they are distributed uniformly, the geometry is not importantnot important
In the brain the Dale principle holds and In the brain the Dale principle holds and the connections are not symmetricthe connections are not symmetric
The hopfield network with assymetric The hopfield network with assymetric weights and dale principle, work properlyweights and dale principle, work properly
Hopfield network and the brain (2):Hopfield network and the brain (2):
The brain is insensitive to noise and local The brain is insensitive to noise and local lesionslesions
Hopfield network can tolerate noise in the Hopfield network can tolerate noise in the input and partial loss of synapsesinput and partial loss of synapses
Hopfield network and the brain (3):Hopfield network and the brain (3):
In brain the neurons are not binary In brain the neurons are not binary devices, they generate continuous values devices, they generate continuous values of firing ratesof firing rates
Hopfield network with sigmoid transfer Hopfield network with sigmoid transfer function is even more powerful than the function is even more powerful than the binary versionbinary version
Hopfield network and the brain (4):Hopfield network and the brain (4):
In the brain most of the neurons are silent In the brain most of the neurons are silent or firing at low rates but in hopfield or firing at low rates but in hopfield network many of the neurons are activenetwork many of the neurons are active
In sparse hopfield network the capacity is In sparse hopfield network the capacity is even moreeven more
Hopfield network and the brain (5):Hopfield network and the brain (5):
In hopfield network updating is serial In hopfield network updating is serial which is far from biological realitywhich is far from biological reality
In parallel updating hopfield network the In parallel updating hopfield network the associative memories can be recalled as associative memories can be recalled as wellwell
Hopfield network and the brain (6):Hopfield network and the brain (6):
When the number of learned patterns in When the number of learned patterns in hopfield network will be overloaded, the hopfield network will be overloaded, the performance of the network will fall performance of the network will fall abruptly for all the stored patterns abruptly for all the stored patterns
But in real brain an overload of memories But in real brain an overload of memories affect only some memories and the rest of affect only some memories and the rest of them will be intactthem will be intactCatastrophic inferenceCatastrophic inference
Hopfield network and the brain (7):Hopfield network and the brain (7):
In hopfield network the usefull information In hopfield network the usefull information appears only when the system is in the appears only when the system is in the stable statestable state
The Brain do not fall in stable states and The Brain do not fall in stable states and remains dynamicremains dynamic
Hopfield network and the brain (8):Hopfield network and the brain (8):
The connectivity in the brain is much less The connectivity in the brain is much less than hopfield networkthan hopfield network
The diluted hopfield network works wellThe diluted hopfield network works well
Hopfield network and the brain (9):Hopfield network and the brain (9):