Biological Modeling of Neural Networks - LCN |...
Transcript of Biological Modeling of Neural Networks - LCN |...
![Page 1: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/1.jpg)
Biological Modeling
of Neural Networks
Week 5
NETWORKS of NEURONS and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
5.1 Introduction
- networks of neuron
- systems for computing
- associative memory
5.2 Classification by similarity
5.3 Detour: Magnetic Materials
5.4 Hopfield Model
5.5 Learning of Associations
5.6 Storage Capacity
Week 5: Networks of Neurons-Introduction
![Page 2: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/2.jpg)
Systems for computing and information processing
Brain Computer
CPU
memory
input
Von Neumann architecture
(10 transistors) 1 CPU
Distributed architecture
10 (10 proc. Elements/neurons)
No separation of
processing and memory
10
![Page 3: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/3.jpg)
10 000 neurons
3 km wire
1mm
Systems for computing and information processing
![Page 4: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/4.jpg)
Brain
Distributed architecture
10 10 neurons
No separation of
processing and memory
4 10 connections/neurons
Systems for computing and information processing
10 000 neurons
3 km wire
1mm
![Page 5: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/5.jpg)
Associations, Associative Memory
Read this text NOW!
I find it rea*l* amazin* t*at y*u ar*
abl* to re*d t*is tex* despit* th* fac*
*hat more t*an t*ent* perc*n* of t**
char*cte*s a*e mis*ing.
*his mean* t*at you* brai* i* abl* ** fill
in missin* info*matio*.
![Page 6: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/6.jpg)
Noisy word
pattern completion/word recognition
brain
List of words
brai*
atom
brave
brain
brass
Output the closest one
Your brain fills in missing information:
‘associative memory’
![Page 7: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/7.jpg)
Biological Modeling
of Neural Networks
Week 5
NETWORKS of NEURONS and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
5.1 Introduction
- networks of neuron
- systems for computing
- associative memory
5.2 Classification by similarity
5.3 Detour: Magnetic Materials
5.4 Hopfield Model
5.5 Learning of Associations
5.6 Storage Capacity
Week 5: Networks of Neurons-Introduction
![Page 8: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/8.jpg)
image
5.2 Classification by similarity: pattern recognition
Classification:
comparison
with prototypes
A B
T
Z
Prototypes Noisy image
![Page 9: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/9.jpg)
Prototypes Noisy image
Classification by closest prototype
AT pxpx
Blackboard:
5.2 Classification by similarity: pattern recognition
![Page 10: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/10.jpg)
Noisy
image
Aim: Understand Associative Memory Associative
memory/
collective
computation
Brain-style computation
Full
image
Partial
word
Full
word
5.2 pattern recognition and Pattern completion
![Page 11: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/11.jpg)
Quiz 5.1: Connectivity
A typical neuron in the brain makes connections
-To 6-20 neighbors
-To 100-200 neurons nearby
-To more than 1000 neurons nearby
-To more than 1000 neurons nearby or far away.
In a typical cristal in nature, each atom interacts
-with 6-20 neighbors
-with 100-200 neurons nearby
-with more than 1000 neurons nearby
-with more than 1000 neurons nearby or far away.
![Page 12: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/12.jpg)
Biological Modeling
of Neural Networks
Week 5
NETWORKS of NEURONS and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
5.1 Introduction
- networks of neuron
- systems for computing
- associative memory
5.2 Classification by similarity
5.3 Detour: Magnetic Materials
5.4 Hopfield Model
5.5 Learning of Associations
5.6 Storage Capacity
Week 5: Networks of Neurons-Introduction
![Page 13: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/13.jpg)
5.3 Detour: magnetism
S
N
![Page 14: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/14.jpg)
Noisy magnet pure magnet
5.3 Detour: magnetism
![Page 15: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/15.jpg)
Elementary magnet
Si = +1
Si = -1 Blackboard:
example
5.3 Detour: magnetism
dynamics
Sum over all
interactions with i
( 1) sgn[ ( )]i j
j
S t S t
![Page 16: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/16.jpg)
Elementary magnet
Si = +1
Si = -1
blackboard
Anti-ferromagnet
wij = +1
wij = -1
5.3 Detour: magnetism
dynamics
Sum over all
interactions with i
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 17: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/17.jpg)
5.3 Magnetism and memory patterns
Elementary pixel
Si = +1
Si = -1
blackboard wij = +1
wij = -1
wij = +1
Hopfield model:
Several patterns next section
dynamics
Sum over all
interactions with i
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 18: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/18.jpg)
Exercise 1: Associative memory (1 pattern)
Elementary pixel
Si = +1
Si = -1
wij = +1 wij = +1
9 neurons
- define appropriate weights
- what happens if one neuron wrong?
- what happens if n neurons wrong?
Next lecture at
10h15
dynamics
Sum over all
interactions with i
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 19: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/19.jpg)
Biological Modeling
of Neural Networks
Week 5
NETWORKS of NEURONS and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
5.1 Introduction
- networks of neuron
- systems for computing
- associative memory
5.2 Classification by similarity
5.3 Detour: Magnetic Materials
5.4 Hopfield Model
5.5 Learning of Associations
5.6 Storage Capacity
Week 5: Networks of Neurons-Introduction
![Page 20: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/20.jpg)
5.4 Hopfield Model of Associative Memory
dynamics
Sum over all
interactions with i
Hopfield model
Prototype
p1
Prototype
p2
jiij ppw
interactions
Sum over all
prototypes
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 21: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/21.jpg)
dynamics
all interactions with i
DEMO
Random patterns, fully connected:
Hopfield model
Prototype
p1
jiij ppw
interactions
Sum over all
prototypes
This rule
is very good
for random
patterns
It does not work well
for correlated patters
5.4 Hopfield Model of Associative Memory
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 22: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/22.jpg)
5.4 Hopfield Model of Associative Memory
jiij ppw
1( ) j jm t S t
Noverlap
1( 1) 1j jm t S t
N
Blackboard
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 23: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/23.jpg)
Hopfield model
Prototype
p1 Finds the closest prototype
i.e. maximal overlap
(similarity) m
Interacting neurons
Computation - without CPU,
- without explicit
memory unit
5.4 Hopfield Model of Associative Memory
![Page 24: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/24.jpg)
Exercise 3 (now)
Prototype
p1 Assume 4 patterns. At time t=0, overlap with
Pattern 3, no overlap with other patterns.
discuss temporal evolution (of overlaps)
(assume that patterns are orthogonal)
1ij i jN
w p p
Sum over all
interactions with i
Next lecture at
11h15
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 25: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/25.jpg)
Biological Modeling
of Neural Networks
Week 5
NETWORKS of NEURONS and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
5.1 Introduction
- networks of neuron
- systems for computing
- associative memory
5.2 Classification by similarity
5.3 Detour: Magnetic Materials
5.4 Hopfield Model
5.5 Learning of Associations
5.6 Storage Capacity
Week 5-5: Learning of Associations
![Page 26: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/26.jpg)
Hebbian Learning
pre
j
post i
ijw
When an axon of cell j repeatedly or persistently
takes part in firing cell i, then j’s efficiency as one
of the cells firing i is increased Hebb, 1949
k
- local rule
- simultaneously active (correlations)
Where do the connections come from?
5.5 Learning of Associations
![Page 27: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/27.jpg)
5.5 Hebbian Learning of Associations
![Page 28: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/28.jpg)
item memorized
5.5 Hebbian Learning of Associations
![Page 29: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/29.jpg)
item recalled
Recall:
Partial info
5.5 Hebbian Learning: Associative Recall
![Page 30: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/30.jpg)
5.5 Associative Recall
Tell me the object shape
for the following list of 5 items: Tell me the color for the following list of 5 items:
be as fast as possible:
time
![Page 31: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/31.jpg)
Tell me the color for the following list of 5 items:
Red
Blue
Yellow
Green
Red
Stroop effect:
Slow response: hard to work
Against natural associations
be as fast as possible:
time
5.5 Associative Recall
![Page 32: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/32.jpg)
Hierarchical organization of
Associative memory
animals
birds fish
Name as fast as possible
an example of a bird swan (or goose or raven or …)
Write down first letter: s for swan or r for raven …
5.5 Associative Recall
![Page 33: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/33.jpg)
name as fast as possible
an example of a hammer
red
Apple
violin
tool
color
music
instrument
fruit
Nommez au plus vite possible
un exemple d’un /d’une
outil
couleur
fruit instrument
de musique
5.5 Associative Recall
![Page 34: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/34.jpg)
Biological Modeling
of Neural Networks
Week 5
NETWORKS of NEURONS and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
5.1 Introduction
- networks of neuron
- systems for computing
- associative memory
5.2 Classification by similarity
5.3 Detour: Magnetic Materials
5.4 Hopfield Model
5.5 Learning of Associations
5.6 Storage Capacity
Week 5-5: Learning of Associations
![Page 35: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/35.jpg)
learning of prototypes
Prototype
p1
Prototype
p2
interactions
Sum over all
prototypes
(1)
Q; How many prototypes can be stored?
dynamics
all interactions with i
jiNij ppw 1
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 36: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/36.jpg)
Prototype
p1
Prototype
p2
jiij ppwInteractions (1)
Q; How many prototypes can be stored?
Dynamics (2)
Random patterns
Minimal condition: pattern is fixed point of dynamics -Assume we start directly in one pattern
-Pattern stays
Attention: Retrieval requires more (pattern completion)
blackboard
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 37: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/37.jpg)
Exercise 4 now: Associative memory Q; How many prototypes can be stored?
Random patterns random walk a) show relation to erf function: importance of p/N
b) network of 1000 neurons – allow at most 1 wrong pixel?
c) network of N neurons – at most 1 promille wrong pixels?
End of lecture, exercise+
Computer exercise : 12:00
Prototype
p1
Prototype
p2
jiij ppwInteractions (1)
Dynamics (2) ( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 38: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/38.jpg)
Prototype
p1
Prototype
p2
jiij ppwInteractions (1)
Dynamics (2)
Random patterns
Minimal condition: pattern is fixed point of dynamics -Assume we start directly in one pattern
-Pattern stays
Attention: Retrieval requires more (pattern completion)
Week 6 Review: storage capacity of Hopfield model
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 39: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/39.jpg)
1 0
errorP
Q; How many prototypes can be stored?
![Page 40: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/40.jpg)
Biological Modeling
of Neural Networks
Week 6
Hebbian LEARNING and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
Week 6: Hopfield model continued
6.1 Stochastic Hopfield Model
6.2. Energy landscape
6.3. Low-activity patterns
6.4. Attractor memorie
- spiking neurons
- experimental data
![Page 41: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/41.jpg)
Prototype
p1
Prototype
p2
interactions
Sum over all
prototypes
(1)
Deterministic dynamics
jiNij ppw 1
6.1 Review: Hopfield model
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 42: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/42.jpg)
Prototype
p1
Prototype
p2
jiij ppwInteractions (1)
Dynamics (2)
Pr{ ( 1) 1| } [ ]i i i j ij jS t h g h g w S t
Random patterns
blackboard Pr{ ( 1) 1| }i i iS t h g p m t
6.1 Stochastic Hopfield model
![Page 43: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/43.jpg)
( )m t t
( )m t0( )m t
6.1 Stochastic Hopfield model: memory retrieval
![Page 44: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/44.jpg)
3 1m17 1m
6.1 Stochastic Hopfield model: memory retrieval
![Page 45: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/45.jpg)
Dynamics (2)
Pr{ ( 1) 1| } [ ]i i i j ij jS t h g h g w S t
blackboard Pr{ ( 1) 1| }i i iS t h g p m t
6.1 Stochastic Hopfield model
Assume that there is only overlap with pattern 17:
two groups of neurons: those that should be ‘on’ and ‘off’ 17Pr{ ( 1) 1| }i iS t h h g m t
17Pr{ ( 1) 1| }i iS t h h g m t
17 17 17 17 172 ( 1) {1 } {1 }m t g m t g m t g m t g m t
![Page 46: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/46.jpg)
( )m t t
( )m t0( )m t
6.1 Stochastic Hopfield model: memory retrieval 17 17 17 17 172 ( 1) {1 } {1 }m t g m t g m t g m t g m t
17 17( 1)m t F m t
![Page 47: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/47.jpg)
Biological Modeling
of Neural Networks
Week 6
Hebbian LEARNING and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
Week 6: Hopfield model continued
6.1 Stochastic Hopfield Model
6.2. Energy landscape
6.3. Low-activity patterns
6.4. Attractor memorie
- spiking neurons
- experimental data
![Page 48: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/48.jpg)
3 1m17 1m
6.2 memory retrieval
![Page 49: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/49.jpg)
E
3 1m17 1m
6.2 Symmetric interactions: Energy picture
![Page 50: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/50.jpg)
Exercise 2 now: Energy picture Next lecture
11:25
E
3 1m17 1m
ij i j
i j
E w S S
( 1) sgn[ ( )]i ij j
j
S t w S t
![Page 51: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/51.jpg)
Biological Modeling
of Neural Networks
Week 6
Hebbian LEARNING and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
Week 6: Hopfield model continued
6.1 Stochastic Hopfield Model
6.2. Energy landscape
6.3. Low-activity patterns
6.4. Attractor memories
- spiking neurons
- experimental data
![Page 52: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/52.jpg)
3 1m17 1m
6.3 Attractor memory
![Page 53: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/53.jpg)
Memory with spiking neurons -Mean activity of patterns?
-Separation of excitation and inhibition?
-Modeling?
-Neural data?
6.3 attractor memory with spiking neurons
![Page 54: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/54.jpg)
1 2 … N i …
=1
=2
=3
jiNij ppw 1
Random patterns +/-1 with zero mean
50 percent of neurons should be active in each pattern
6.3 attractor memory with low activity patterns
![Page 55: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/55.jpg)
1 2 … N i …
=1
=2
=3
1 ( )( )ij i jNw p b p a
Random patterns +/-1 with low activity (mean =a<0)
20 percent of neurons should be active in each pattern
activity Some constant
![Page 56: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/56.jpg)
Biological Modeling
of Neural Networks
Week 6
Hebbian LEARNING and
ASSOCIATIVE MEMORY
Wulfram Gerstner
EPFL, Lausanne, Switzerland
Week 6: Hopfield model continued
6.1 Stochastic Hopfield Model
6.2. Energy landscape
6.3. Low-activity patterns
6.4. Attractor memories
- spiking neurons
- experimental data
![Page 57: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/57.jpg)
Inh1
Inh2
theta
Exc
1 ( 1)( 1)ij i jNw p p
Inh1
Inh2 Hebb-rule:
Active together
6.4 attractor memory with spiking neurons
![Page 58: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/58.jpg)
Spike raster
Overlap with patterns 1 … 3
![Page 59: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/59.jpg)
overlaps
Spike raster
Overlap with patterns 1 … 11 (80 patterns stored!)
![Page 60: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/60.jpg)
Memory with spiking neurons -Low activity of patterns?
-Separation of excitation and inhibition?
-Modeling?
-Neural data?
All possible
![Page 61: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/61.jpg)
Sidney
opera Sidney
opera
Sidney
opera
Human Hippocampus
Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C., and Fried, I. (2005).
Invariant visual representation by single neurons in the human brain.
Nature, 435:1102-1107.
6.4 memory data
![Page 62: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/62.jpg)
Delayed Matching to Sample Task
1s sample match
1s sample match
Animal experiments
6.4 memory data
![Page 63: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/63.jpg)
20
1s sample match
[Hz]
Miyashita, Y. (1988). Neuronal correlate of visual associative long-term
memory in the primate temporal cortex. Nature, 335:817-820.
6.4 memory data
![Page 64: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/64.jpg)
match
20
[Hz]
sample
0 1650ms
0
Rainer and Miller (2002). Timecourse of object-related neural activity in the primate
prefrontal cortex during a short-term memory task. Europ. J. Neurosci., 15:1244-1254.
6.4 memory data
![Page 65: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/65.jpg)
In the Hopfield model, neurons are characterized by a binary variable
Si = +/-1. For an interpretation in terms of spikes it is, however, more
appealing to work with a binary variable xi which is zero or 1.
(i) Write Si = 2xi- 1 and rewrite the Hopfield model in terms of xi.
What are the conditions so that the input potential is
(ii) Repeat the same calculation for low-activity patterns and
and weights
with some constants a and b
1 ( )( )ij i jNw p b p a
i j ij jh w x
Exercise 3 NOW- from Hopfield to spikes
![Page 66: Biological Modeling of Neural Networks - LCN | EPFLlcn.epfl.ch/~gerstner/NeuronalDynamics-Classes/week5-Hopfield.pdf · 5.4 Hopfield Model 5 ... Biological Modeling of Neural Networks](https://reader033.fdocuments.us/reader033/viewer/2022050714/5ba4103209d3f280548c9d03/html5/thumbnails/66.jpg)
The end