Prapun B-Exam
Transcript of Prapun B-Exam
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Capacity Analysis of Neurons with DescendingAction Potential Thresholds
Prapun Suksompong
Electrical and Computer EngineeringCornell University, Ithaca, NY 14853
Final Examination for the Doctoral Degree (“B” Exam)July 24, 2008
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Outline
Introduction
Sources of Variability for the ISIs and Derivation of the Threshold
Information-Theoretic Analysis of IF Neuron
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
IntroductionNeuron MorphologyIntegrate-and-Fire NeuronsGoal
Sources of Variability for the ISIs and Derivation of the Threshold
Information-Theoretic Analysis of IF Neuron
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Neuron Morphology
A neuron is the basic working unit of the nervous system.
A typical neuron has threefunctionally distinct parts, called
I dendrites,
I soma, and
I axon.
The junction between twoneurons is called a synapse.
Nucleus
Axon
Dendrite
Axon Terminals
Myelin Sheath
Cell Body (Soma)
Presynaptic Axom
Terminal Synaptic Cleft
Postsynaptic Dendrite
Ion Channel
Synapse
Axon from another neuron
Node of Ranvier
Synaptic Vesicle
Axon Hillock
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Action Potentials (Spikes)
Looking at a synapse, we refer to the sending neuron as thepresynaptic neuron and to the receiving neuron as thepostsynaptic neuron.
postsynaptic presynaptic
axon synapse
The neuronal signals consist of short electrical pulses called actionpotentials (APs) or spikes. A chain of APs emitted by a singleneuron is called a spike train.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Quantal Synaptic Failure (QSF)
postsynaptic presynaptic
axon synapse
I Synaptic failure: It is possible that an AP fails to get“across” the synapse.
I We may model a synapse as a Z -channel.
I Spikes which successfully cross the synapse then propagatedown to soma.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Integrate-and-Fire Neurons
⊕
I Assumption: ∼ 104
pre-synaptic neurons.
I True in cortex (higher brainfunctions).
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Integrate-and-Fire Neurons
⊕
I Spikes generated when themembrane potentials hit thethresholds.
I Descending thresholds.
Thank you for coming to this talk. This work is a joint effort between Prof. Berger and me. We also got quite some help from Dr. Levy over here. For this session, we will focus on the timing jitter which occurs in communication between neuron. First, let’s recall that a neuron receives spike trains from many neurons and for us we will assume that the number of incoming connections is large, say, on the order of ten-thousand. The neuron in the middle of the figure sums up the contribution from each incoming spikes and when its membrane potential reaches a specific value which everyone call the “threshold”, we get a spike which propagate to another neuron. Now, instead of a constant threshold whose value is fixed at a specific level, the thresholds which are drawn here are in fact decreasing. We will return to this later.
Descending Threshold
Ascending Membrane Potential
Spike Train time
time
• First jitter: Spike generation • Poisson approximation
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
(Leaky) Integrate-and-Fire Model: LIF or IF
Let τ1, τ2, τ3, . . . be the sequence of time that the spikes arrive atthe spike generating region. The membrane potential at time t isthen
X (t) =∑m
h (t, τm,Ym) =∑m
Ymh(t − τm).
This is the “integrate” part of the integrate-and-fire neuron.
I Ym is the weight for the mth
spike due to propagationloss, synaptic strength,synaptic failure, etc.
I h is the shape function.
( )X t
( )2 2i iY h t τ+ +−
( )1 1i iY h t
( )i iY h t −τ
1iτ + 2iτ +
τ+ +−
i
time τ
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Integrate-and-Fire Neurons (Con’t)
I Constant bombardment of spikesleads to increase in membranepotential.
I As soon as the membrane potentialreaches a critical value orthreshold, the neuron “fires” anaction potential. Then, everythingresets.
I Refractory period: The time aftera AP is produced, during which itis impossible to generate anotherAP.
I Set T (t) to be ∞ during thisperiod.
time (t)
time (t)
Threshold
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Integrate-and-Fire Neurons (Summary)
I ∼ 104 pre-synaptic neurons.
⊕
Thank you for coming to this talk. This work is a joint effort between Prof. Berger and me. We also got quite some help from Dr. Levy over here. For this session, we will focus on the timing jitter which occurs in communication between neuron. First, let’s recall that a neuron receives spike trains from many neurons and for us we will assume that the number of incoming connections is large, say, on the order of ten-thousand. The neuron in the middle of the figure sums up the contribution from each incoming spikes and when its membrane potential reaches a specific value which everyone call the “threshold”, we get a spike which propagate to another neuron. Now, instead of a constant threshold whose value is fixed at a specific level, the thresholds which are drawn here are in fact decreasing. We will return to this later.
Descending Threshold
Ascending Membrane Potential
Spike Train time
time
• First jitter: Spike generation • Poisson approximation
I Descending thresholds.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Theoretical Approaches to Neuroscience
I We use IF model, but more biologically-realistic models exist(e.g. Hodgkin and Huxley [’52] model).
1. Too many parameters.I Physical measurements “fundamentally disturb cell properties”
2. Provide less insight.
I Biological structures have evolved via natural selection tooperate optimally.
I See the book Optima for Animals by R. McNeill Alexander.I What is the best strength for a bone?I At what speed should humans change from walking to
running?
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Information-Theoretic Optimization
Application of information theory has already found success inmany areas of neuroscience.
I Barlow’s “economy of impulses”[’59, ’69]I Minimize redundancy.
I Linsker’s InfoMax principle [’88, ’89]I Maximize the mutual information.
I Levy and Baxter’s energy-efficient coding [’96, ’02]I Maximize mutual information per unit energy expended.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Neuron MorphologyIntegrate-and-Fire NeuronsGoal
Motivation and Goal
I Integrate-and-Fire (IF) model is very popular.
I The threshold function is a crucial element of the IF model.
I Little amount of work exists on deriving the form of thethreshold curve.
I In fact, using constant thresholding is also popular.I This leads to large jitter in the spike timing and hence
discourages the use of time coding.
Goal: Find (1) an expression for threshold curve under biologicallyrealistic constraints and (2) the optimal operating point of neuronunder such threshold.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Introduction
Sources of Variability for the ISIs and Derivation of the ThresholdFirst Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Information-Theoretic Analysis of IF Neuron
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Three sources of variability for inter-spike intervals
1. Spike generation
2. Spike propagation
3. Time-of-arrival estimation
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
First jitter: Spike generationformula that governs how the membrane potential of this middle neuron rises given the combined incoming rate λ .
⊕
1λ
2λ
3λ
21 3λ λλλ= + + +
⊕
Now, of course, there is some jitter in the timing of the spike and that’s the focus of our paper today. The first source of jitter comes from the fact that the incoming spike trains on this side has in fact some jitter in them. Now, you may recall that we assume that number of these presynaptic neurons are large. That assumption allows us to say that the combined effect .. the superposed spike trains … is close to a Poisson process even though the individual spike train coming out of a single neuron is not a Poisson process. That allows us to find a tractable formula that governs how the membrane potential of this middle neuron rises given the combined incoming rate λ .
I Large number of presynaptic neurons allows Poissonapproximation for the superposed process.
I The membrane potential is governed by a filtered Poissonprocess.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Approximation for the first timing jitter
I For fixed λ, different realizations of the membrane potentialcorrespond to different spiking times.
Xσ
Membrane potential ( )X t
Threshold ( )T t
timeσ Time
I Filtered Poissonapproximation foramount of variationin vertical direction[Parzen’62].
I Linear approximationfor amount ofvariation in horizontaldirection.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Approximation for the first timing jitter (con’t)
σtime (τ) ≈ H (τ)
T (τ) h (τ)− T ′ (τ) H (τ)
√T (τ) c2H2 (τ)
c1H (τ)
Xσ
Membrane potential ( )X t
Threshold ( )T t
timeσ Time
I h : shape function.I e.g. exponential
Xσ
Membrane potential ( )X t
Threshold ( )T t
timeσ Time
The figure here shows different realizations of the membrane potentials for a fixed combined incoming rate λ . Here, we see that the randomness from the Poisson arrivals causes fluctuation in the time that the membrane potentials hit the threshold. Under some linear approximation, we can relate the jitter in the vertical direction to the one in horizontal direction. This then gives us the formula for the magnitude of the timing jitter as a function of the spike time τ .
Here, h is the shape function which describes how the membrane potential changes in response to a single input spike. For the usual leaky integrate-and-fire model, this h starts with some amplitude and then decay exponentially.
( )h t
For conciseness, we define these two integrations which get used in the formula here.
I H (t) =
t∫0
h (µ)dµ.
I H2 (t) =t∫
0
h2 (µ)dµ.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Approximation for the first timing jitter (con’t)
σtime (τ) ≈ H (τ)
T (τ) h (τ)− T ′ (τ) H (τ)
√T (τ) c2H2 (τ)
c1H (τ)
Xσ
Membrane potential ( )X t
Threshold ( )T t
timeσ Time
I c1 and c2 are constantswhich depend on thedistribution of the weight(Ym) for each spike.
Recall:X (t) =
∑m Ymh(t − τm).
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Jitter in rate estimation
I The only information contained in a Poisson process is its rateλ.
I Different λ’s ⇒ different spiking times τ ’s.
PSP for Large λ
PSP for Small λ
Descending Threshold
Time
However, this spike time has some jitter, so the $\lambda$ estimation also have some error. We then go on and approximate this error:
We also consider two other sources of jitters which I won’t go into details. The second jitter is the randomness in the length of time a spike takes to propagate to a synapse on a particular neuron. IT is on the order of 10 microseconds. The third jitter is the time-of-arrival estimation error.
I Spike times vary inversely with λ.
λ(τ) ≈ T (τ)
c1H (τ)
I Error in rate estimation:
σλ (τ) ≈ 1
c1H (τ)
√T (τ) c2H2 (τ)
c1H (τ).
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Second and Third Jitters
I Propagation Time.
I Time-of-Arrival EstimationError (radar rangingproblem):(
4π2 2Es
N0f 2
)−1
.
I
√f 2: Gabor-bandwidth.
I ES : Signal energy.I N0
2 : Spectral height of theNoise.
I Small: < 10µs.
For the together two sources of jitter. I won’t go into details. The second jitter is the randomness in the length of time a spike takes to propagate to a synapse on another neuron. It is on the order of 10 microseconds. The third jitter is the time-of-arrival estimation error; that is, if this neuron tries to measure the inter-spike interval, it needs to find out what time a spike arrives. We borrow some formula from the Radar guys shown here because this is exactly the problem that they call the radar ranging problem. The error depends on the signal-to-noise ratio which is shown here as this Es over N0 here. It also depends on the bandwidth for the shape of the action potential. The amount of error here is about 10 microseconds as well.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Deriving The Threshold Curves
Recall: Deviation in time:
σtime (τ) =H (τ)
T (τ) h (τ)− T ′ (τ) H (τ)
√T (τ) c2H2 (τ)
c1H (τ).
We consider the thresholds which
1) preserve timing jitter σtime (τ) ≡ σtime,0, or
2) preserve relative timing jitter σtime(τ)τ ≡ σ%time,0
across spiking times (or spiking frequencies) of interest.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Deriving The Threshold Curves
Recall:
(a) Deviation in time: σtime (τ) = H(τ)T (τ)h(τ)−T ′(τ)H(τ)
√T (τ)c2H2(τ)
c1H(τ) .
(b) Deviation in λ estimation: σλ (τ) = 1c1H(τ)
√T (τ)c2H2(τ)
c1H(τ) .
We consider the thresholds which
1) preserve timing jitter σtime (τ) ≡ σtime,0, or
2) preserve relative timing jitter σtime(τ)τ ≡ σ%time,0, or
3) preserve jitter in λ estimation σλ (τ) ≡ σλ,0, or
4) preserve relative jitter in λ estimation σλ(τ)λ ≡ σ%λ,0, or
5) preserve jitter in lnλ estimation
across spiking times (or spiking frequencies) of interest.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Deriving The Threshold Curves (con’t)
I Constant timing jitter level:
T ′ (t) = T (t)
(h (t)
H (t)
)−√
T (t)
(1
σtime,0
√c2H2 (t)
c1H (t)
).
I Constant relative-timing-jitter level:
T ′ (t) = T (t)
(h (t)
H (t)
)−√
T (t)
(1
tσ%time,0
√c2H2 (t)
c1H (t)
).
I Preserve relative error in λ estimation:
T (t) = c0H2 (t)
H (t).
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
The differential equations are Bernoulli equations of the form
T ′ (t) = T (t) P (t)−√
T (t)Q (t) .
They can be reduced to linear equation by introducingv(t) =
√T (t) which gives
v ′ (t) =1
2P (t) v (t)− 1
2Q (t) .
Linear! The solution is
v (t) = v (t0)φ (t, t0)−t∫
t0
1
2φ (t, τ) Q (τ)dτ,
where φ (t, s) = e
t∫s
12P(τ)dτ
.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Comparison between derived thresholds
6 7 8Time [ms]
Linear
Constant timing jitter
Constant relative timing jitter
Exponential
Heavy-tail
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
First Jitter: Spike generationSecond and Third JittersThe Threshold Curve
Summary
I Analyze and quantify three sources of timing jitter
I Predict shape of threshold curves
6 7 8Time [ms]
Linear
Constant timing jitter
Constant relative timing jitter
Exponential
Heavy-tail
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Introduction
Sources of Variability for the ISIs and Derivation of the Threshold
Information-Theoretic Analysis of IF NeuronOPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Conditional density Q(t|λ) = fτ |Λ(t|λ)
I We have formula(s) for the threshold curve T (t).
I Assumption: λ stays constant during each ISI.
I Given Poisson input intensity λ, can find the conditionaldensity Q(t|λ) = fτ |Λ(t|λ).
I τ = g(Λ)+jitter.
τ Λ λ
τ|Λ |λ
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Optimization 1: Mutual Information
OPT1:sup I (Λ; τ)
where
I (Λ; τ) = E[
logfΛ,τ (Λ, τ)
fΛ(Λ)fτ (τ)
]and the supremum is taken over all possible fΛ(λ).
I Blahut-Arimoto Algorithm (BAA)
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Exponential Threshold
“Constant-Jitter” Threshold
“Constant-Relative-Jitter” Threshold
C = 5.457 C = 4.931 C = 5.109
(a) (b) (c)
Exponential Threshold
“Constant-Jitter” Threshold
“Constant-Relative-Jitter” Threshold
C = 5.4743 C = 5.2558 C = 4.8279
(a) (b) (c)
Algorithm Exponential Threshold
“Constant Jitter” Threshold
“Constant Relative Jitter” Threshold
Blahut-Arimoto 5.4743 5.2558 4.8279
0 500 1000 1500 20000
2
4
6x 10
-3
f (
)
4 6 8 10 12 140
0.2
0.4
0.6
0.8
t
f (t
)
0 500 1000 1500 20000
2
4
6x 10
-3
f
()
4 5 6 7 80
0.2
0.4
0.6
0.8
t
f (t
)0 500 1000 1500 2000
0
2
4
6x 10
-3
f (
)
4 5 6 7 8 90
0.2
0.4
0.6
0.8
t
f (t
)
0 200 400 600 800 1000 1200 1400 1600 1800 20000
0.005
0.01
0.015
0.02
f (
)
2 4 6 8 10 12 14 160
0.2
0.4
0.6
0.8
f ()
0 200 400 600 800 1000 1200 1400 1600 1800 20000
0.01
0.02
0.03
f (
)
4.5 5 5.5 6 6.5 7 7.5 80
0.2
0.4
0.6
0.8
f ()
0 200 400 600 800 1000 1200 1400 1600 1800 20000
0.01
0.02
0.03
f (
)
4.5 5 5.5 6 6.5 7 7.50
0.2
0.4
0.6
0.8
f ()
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Capacity-achieving input densities look similar.
0 200 400 600 800 1000 1200 1400 1600 1800 20000
1
2
3
4
5
6x 10
−3
λ
f Λ(λ
)
exponentialconstant jitterconstant relative jitter
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Input-Intensity Density Approximation
I BAA does not provide anyinsight.
Our simpler formula:
fΛ (λ) ≈ σ0
d
√(c1H (t))3
T (t) c2H2 (t)
∣∣∣∣∣∣t=g(λ)
where g(λ) = E [τ |λ].
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Input-Intensity Density Approximation
I BAA does not provide anyinsight.
Our simpler formula:
fΛ (λ) ≈ σ0
d
√(c1H (t))3
T (t) c2H2 (t)
∣∣∣∣∣∣t=g(λ)
where g(λ) = E [τ |λ].0 200 400 600 800 1000 1200 1400 1600 1800 2000
0
1
2
3
4
5
6x 10
−3
λ
f Λ(λ
)
exponentialconstant jitterconstant relative jitterapproximation
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Input-Intensity Density Approximation
I BAA does not provide anyinsight.
Our simpler formula:
fΛ (λ) ≈ σ0
d
√(c1H (t))3
T (t) c2H2 (t)
∣∣∣∣∣∣t=g(λ)
where g(λ) = E [τ |λ]. 101
102
103
10−3
λ
f Λ(λ
)
exponentialconstant jitterconstant relative jitterapproximation
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Approximation Strategy
Assume that g(Λ) is uniform and then find the corresponding fΛ.
Exponential Threshold
“Constant-Jitter” Threshold
“Constant-Relative-Jitter” Threshold
C = 5.457 C = 4.931 C = 5.109
(a) (b) (c)
Exponential Threshold
“Constant-Jitter” Threshold
“Constant-Relative-Jitter” Threshold
C = 5.4743 C = 5.2558 C = 4.8279
(a) (b) (c)
Algorithm Exponential Threshold
“Constant Jitter” Threshold
“Constant Relative Jitter” Threshold
Blahut-Arimoto 5.4743 5.2558 4.8279
0 500 1000 1500 20000
2
4
6x 10
-3
f (
)
4 6 8 10 12 140
0.2
0.4
0.6
0.8
t
f (t
)
0 500 1000 1500 20000
2
4
6x 10
-3
f (
)
4 5 6 7 80
0.2
0.4
0.6
0.8
t
f (t
)
0 500 1000 1500 20000
2
4
6x 10
-3
f (
)
4 5 6 7 8 90
0.2
0.4
0.6
0.8
t
f (t
)
0 200 400 600 800 1000 1200 1400 1600 1800 20000
0.005
0.01
0.015
0.02
f (
)
2 4 6 8 10 12 14 160
0.2
0.4
0.6
0.8
f ()
0 200 400 600 800 1000 1200 1400 1600 1800 20000
0.01
0.02
0.03
f (
)
4.5 5 5.5 6 6.5 7 7.5 80
0.2
0.4
0.6
0.8
f ()
0 200 400 600 800 1000 1200 1400 1600 1800 20000
0.01
0.02
0.03
f (
)
4.5 5 5.5 6 6.5 7 7.50
0.2
0.4
0.6
0.8
f ()
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Approximation Strategy
Assume that g(Λ) is uniform and then find the corresponding fΛ.
*
𝑓 𝑔
𝑓𝑔
𝑓𝑁 𝑓
Convolution
For invertible function g , the pdf of Z = g(Λ) is given by
fZ (z) =
∣∣∣∣ d
dzg−1 (z)
∣∣∣∣ fΛ (g−1 (z))
=1
|g ′ (λ)|fΛ (λ) ,
where z = g(λ).Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
OPT2: Energy-Efficient Neuron
I Brains consume 20% of energy consumption for adults and60% for infant [Laughlin and Sejnowski’03].
I Suppose neuron spendsI 1 unit of energy per ms when it is idle, andI e unit of energy per ms when AP is produced.
I e >> 1.
I If the time to the next spike is τ = t, the energy expended is
bo(t) = 1× (t −∆) + e ×∆ = t + (e − 1)∆ = t + r .
where ∆ is the time used to produce a spike.I The value of r depends on the type of neurons under
consideration.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Optimization 2: I/E
OPT2:
supI (Λ; τ)
E [bo(τ)]
where the supremum is taken over all possible fΛ(λ).
I Jimbo-Kunisawa algorithm (JKA) maximizes I (Λ;τ)E[b(Λ)] .
I b is a function of input.
I Our bo is a function of output.I We define b(λ) = E [bo(τ)|Λ = λ] and apply JKA.
I Because bo(τ) = τ + r , we have
b(λ) = E [τ |λ] + r = g(λ) + r .
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Input-Intensity Density Approximation
I Can use the same technique as in OPT1 to do approximationof input-intensity density.
I In stead of uniform density, consider bounded exponentialdensity of the form
f (t; γ, α, β) =γ
e−γα − e−γβe−γt1[α,β] (t) .
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Input-Intensity Density Approximation
I Can use the same technique as in OPT1 to do approximationof input-intensity density.
I Result:
fΛ (λ) ≈ σ0f (t; γ, g(b), g(a))
√(c1H (t))3
T (t) c2H2 (t)
∣∣∣∣∣∣t=g(λ)
,
where f (t; γ, g(b), g(a)) is the bounded exponential pdf withsupport on the interval [g(b), g(a)] and parameter γ.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
0 500 1000 1500 20000
1
2
3
4
5
6x 10
−3
λ
f Λ(λ
)
exponentialconstant jitterconstant relative jitterapproximation
4 6 8 10 12 140
0.1
0.2
0.3
0.4
0.5
t [ms]
f τ(t)
exponentialconstant jitterconstant relative jitter
101
102
103
10−3
λ
f Λ(λ
)
exponentialconstant jitterconstant relative jitterapproximation
4 5 6 7 8 9 10
10−1
t [ms]
f τ(t)
exponentialconstant jitterconstant relative jitter
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Free Parameters - Revisited
Recall, for example, the differential equation that define our“constant-relative-jitter” threshold:
T ′ (t) = T (t)
(h (t)
H (t)
)−√
T (t)
(1
tσ%time,0
√c2H2 (t)
c1H (t)
).
There are a couple of parameters which we want to revisit.I The constants c1 and c2.
I Embedded in them is the effect of QSF.
I σ%time,0 =σtime,0
t0. What value should we set σtime,0 to be?
I > 10µs.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
I By scaling the unit of the voltage, we can make c1 = 1.
I The scaling makes
c2 ∝1
psuccess=
1
1− pfailure
where pfailure is the QSF probability.I pfailure depends on the type of neurons under consideration.
I Let σtime,0 = σ1 and play with it.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Average Rates
From our optimization (either OPT1 or OPT2), we get the optimalinput-intensity density fΛ(λ) and spiking-time density fτ (t) for eachvalue of σ1.
I Each value of σ1 gives average arrival rate λ̄in and the averagespiking rate λ̄out.
I λ̄out = 1E[τ ] .
I λ̄in = E [Λ]?
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Average Afferent Rate
I High value of Λ corresponds to low value of τ .
I Neuron experiences large Λ value for short amount of time.
I λ̄in should be lower than E [Λ].
I Let (Λi , τi ) be the pair of input-intensity and the length of theISI associated with the ith spike.
I The number of input spikes during the ith ISI is a Poissonrandom variable Ni with mean Λiτi .
I The long-term average input rate is then∑ki=1 Ni∑ki=1 τi
=1k
∑ki=1 Ni
1k
∑ki=1 τi
→ E [Λ1τ1]
E [τ1].
I λ̄in = E[Λτ ]E[τ ] .
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Rate Matching
I On average, our neuron in consideration is bombarded with anaverage input-intensity λ̄in.
I Suppose our neuron is receiving input from n other neurons.Then, on average, each of the sending neurons fire at a rateof 1
n λ̄in.
I Our neuron should also generate spikes at this rate.
I Therefore, we must have
1
nλ̄in = λ̄out.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Rate Matching (Con’t)
10 20 30 40 50 60 70 80 90 1000
50
100
150
200
σ1 [microsec]
Ave
rage
spi
king
rat
e [s
pike
s/s]
inputoutput
10 20 30 40 50 60 70 80 90 100
4.8
5
5.2
5.4
σ1 [microsec]
Cap
acity
[bits
]
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Rate Matching (Con’t)
I Capacity value increases aswe increase the noise levelσ1.
I Larger value of σ1 impliesthreshold decays slower Thisgives larger support for thespiking time.
I Noise is small. Effect ofincreasing the support of theoutput is stronger than theeffect of increased noise.
10 15 20 25 30 35 40 45 50 55 600
50
100
150
200
σ1 [microsec]
Ave
rage
spi
king
rat
e [s
pike
s/s]
inputoutput
10 15 20 25 30 35 40 45 50 55 60
5.2
5.4
5.6
5.8
σ1 [microsec]
Cap
acity
[bits
]
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
1 1.5 2 2.5 3 3.5 4 4.5 515
20
25
30
35
40
45
50
55
60
c2
Opt
imal
rate
[spi
kes
per s
econ
d]
(c) max I/E, r = 100
exponentialconstant relative jitter
(b) max I/E, r = 400
(a) max I
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Rate Matching (Con’t)
I Firing rate decreases as thefailure probability increases.
I Smaller value of rcorresponds to higher rate.
I As r →∞, the rateconverges to the max-I case.
1 1.5 2 2.5 3 3.5 4 4.5 515
20
25
30
35
40
45
50
55
60
c2
Opt
imal
rate
[spi
kes
per s
econ
d]
(c) max I/E, r = 100
exponentialconstant relative jitter
(b) max I/E, r = 400
(a) max I
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
OPT1: Maximization of Mutual InformationOPT2: Mutual Information per Unit Energy CostRate Matching
Optimal Threshold: c2 = 5 and r = 400
0 20 40 60 80 100 120 140 160 180 2000
100
200
300
400
500
600
700
800
Time [ms]
exponentialconstant relative timing jitter
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Introduction
Sources of Variability for the ISIs and Derivation of the Threshold
Information-Theoretic Analysis of IF Neuron
Conclusion
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
Contributions
1. Quantify the amount of timing jitter in neuron.
2. Construct threshold functions.
3. Provide optimal operating points for neurons which are closeto experimentally observed values.
I Formulas to approximate the optimal input-intensity densities.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
References I
P. Suksompong and T. Berger.Jitter Analysis of Timing Codes for Neurons with DescendingAction Potential Thresholds.ISIT, 2006.
P. Suksompong and T. Berger.Capacity Analysis of Neurons with Descending ActionPotential Thresholds.In preparation for special issue of IEEE Tran. on Info. Theory,2009.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
References II
P. Suksompong and T. Berger.Energy-Efficient Neurons with Descending Action PotentialThresholds.In preparation for Journal of Comp. Neuroscience, 2009.
T. Berger and W.B. Levy.Encoding of excitation via dynamic thresholding.Society for Neuroscience, 2004.
W.B. Levy and R.A. Baxter.Energy-Efficient Neuronal Computation via Quantal SynapticFailures.Journal of Neuroscience, 2002.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
References III
W.B. Levy and R.A. Baxter.Energy Efficient Neural CodesNeural Computation, 1996.
Patrick Crotty and William Levy.Biophysical limits on axonal transmission rates in axons.CNS, 2005.
E. Parzen.Stochastic Processes.Holden Day, 1962.
H. Vincent Poor.An introduction to signal detection and estimation (2nd ed.).Springer-Verlag New York, Inc., New York, NY, USA, 1994.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
References IV
Dale Purve et al.Neuroscience (3rd ed.).Sinauer Associates Inc., Sunderland, MA USA, 1997.
Prapun Suksompong Capacity Analysis of Neurons
IntroductionSources of Variability and Threshold Derivation
Information-Theoretic Analysis of IF NeuronConclusion
THE END
Prapun Suksompong Capacity Analysis of Neurons