Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic...

34
Dynamic Boltzmann Machine IBM Research – Tokyo Takayuki Osogami [email protected]

Transcript of Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic...

Page 1: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Dynamic Boltzmann Machine

IBM Research – Tokyo

Takayuki Osogami

[email protected]

Page 2: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

How can we make effective use of spike-timing dependent plasticity (STDP) in artificial neural networks?

Hebb’s rule (’49) STDP (’90s)Cells that fire together, wire together

Bi & Poo (1998)Dan & Poo (2006)

Amount of changes depends on timing of spikes

Today’s artificial neural networks ?[Nessler et al. 2013, Bengio et al. 2016, Scellier & Bengio 2016]

Page 3: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Takayuki Osogami and Makoto Otsuka, “Seven neurons memorizing sequences of alphabetical images via spike-timing dependent plasticity,” Scientific Reports, 5, 14149 (2015). www.nature.com/articles/srep14149

Page 4: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

This talk

• Boltzmann machine, Hebb’s rule, STDP

• Dynamic Boltzmann machine [O & Otsuka (2015)]

• Experiments [O & Otsuka (2015), Dasgupta & O (2017)]

Page 5: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Boltzmann machine

xi

xjwij

Energy of being x:𝐸 𝒙 = −𝒙⊤𝑾𝒙

Probability of being x:

𝑃 𝒙 =1

𝑍exp −𝐸 𝒙

Parameters: 𝑾 = (𝑤𝑖𝑗)

Variables: x = (x1,x2,…)𝑍 ≡

𝒙

exp(−𝐸 𝒙 )

Page 6: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Learning rule of Boltzmann machine, maximizing log-likelihood [Hinton et al. ’83]

Neuron i

xi

Neuron j

xj

Synapsewij

𝑤𝑖𝑗 ← 𝑤𝑖𝑗 + 𝜂 𝑥𝑖 𝑥𝑗− 𝑋𝑖 𝑋𝑗

𝜕

𝜕𝑤𝑖𝑗𝐿𝐿 𝐷 =

𝒙∈𝐷

𝑥𝑖 𝑥𝑗− 𝑋𝑖 𝑋𝑗

Expected value:

𝑋𝑖 𝑋𝑗 ≡

𝒙

𝑃 𝒙 𝑥𝑖 𝑥𝑗

Log likelihood of training data D:

𝐿𝐿 𝐷 =

𝒙∈𝐷

log(𝑃 𝒙 )

cf. Hebb’s rule

Stochastic gradient

Page 7: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Do not distribute

Pre-synaptic neuron

Post-synaptic neuron

Image courtesy of dream designs at FreeDigitalPhotos.net

Page 8: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Spike-timing dependent plasticity (STDP): Amount of changes depends on timing of spikes

Synapse strengthened(Long Term Potentiation)

Bi & Poo (1998)Dan & Poo (2006)

Pre-synapticneuron i

xi

Post-synapticneuron j

xj

Synapsewij

Synapse weakened(Long Term Depression)

Page 9: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

We will construct a dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines

x[0]x[-1]x[-T]

Time

𝑇 → ∞ Dynamic Boltzmann machineBoltzmann machine for a T-th order Markov model

Historical values Next value

Particular parametric form of weight

Page 10: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

This talk

• Boltzmann machine, Hebb’s rule, STDP

• Dynamic Boltzmann machine (DyBM) [O & Otsuka (2015)]

• DyBM with LTP

• DyBM with LTP and LTD

• Experiments [O & Otsuka (2015), Dasgupta & O (2017)]

Page 11: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

LTP: Weight is strengthened when the post-synaptic neuron fires shortly after the pre-synaptic neuron

Synapse strengthened(Long Term Potentiation)

Pre-synapticneuron i

xi

Post-synapticneuron j

xj

Synapsewij

Bi & Poo (1998)Dan & Poo (2006)

Page 12: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines

x[0]x[-1]x[-T]

Time

𝑇 → ∞ Dynamic Boltzmann machineBoltzmann machine for a T-th order Markov model

Historical values Next value

Weight from neuron i at time t to neuron j at time t + 𝛿

𝑤𝑖𝑗𝛿= 𝑢𝑖𝑗 𝜆

𝛿−𝑑𝑖𝑗

𝛿𝑑𝑖𝑗

We learn 𝑢𝑖𝑗

Page 13: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Dynamic Boltzmann machine (LTP only)

xi[t] xj

[t]xi[t-1]

Conduction delay, dij

Synaptic eligibility trace:

𝛼𝑖𝑗[𝑡−1]

𝑠<𝑡−𝑑𝑖𝑗

𝜆𝑡−𝑠−𝑑𝑖𝑗 𝑥𝑖𝑠

𝛼𝑖𝑗[𝑡]

← 𝜆 𝛼𝑖𝑗𝑡−1

+ 𝑥𝑖

𝑡−𝑑𝑖𝑗

xi[t-2] xi

[t-3]Probability for neuron j to fire at time t:

𝑋𝑗𝑡

≡ 𝑃 𝑥𝑗𝑡= 1|𝑥 :𝑡−1

=1

1 + exp − 𝑖 𝑢𝑖𝑗 𝛼𝑖𝑗𝑡−1

Page 14: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Learning rule of DyBM, maximizing log-likelihood𝜕

𝜕𝑢𝑖𝑗log 𝑃 𝑥 𝑡 𝑥[:𝑡−1] = 𝛼𝑖𝑗

𝑡−1𝑥𝑗

𝑡− 𝑋𝑗

𝑡

Stochastic gradient for LTP weight:

𝑢𝑖𝑗 ← 𝑢𝑖𝑗 + 𝜂 𝛼𝑖𝑗𝑡−1

𝑥𝑗𝑡− 𝑋𝑗

𝑡

Spike-timing dependent

How recently/often spikes reached from neuron i

xi[t] xj

[t]xi[t-1]

Conduction delay, dij

xi[t-2] xi

[t-3]

cf. Boltzmann machine𝑤𝑖𝑗 ← 𝑤𝑖𝑗 + 𝜂(𝑥𝑖 𝑥𝑗− 𝑋𝑖 𝑋𝑗 )

Synaptic eligibility trace:

𝛼𝑖𝑗[𝑡−1]

𝑠<𝑡−𝑑𝑖𝑗

𝜆𝑡−𝑠−𝑑𝑖𝑗 𝑥𝑖𝑠

𝛼𝑖𝑗[𝑡]

← 𝜆 𝛼𝑖𝑗𝑡−1

+ 𝑥𝑖

𝑡−𝑑𝑖𝑗

Page 15: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

This talk

• Boltzmann machine, Hebb’s rule, STDP

• Dynamic Boltzmann machine (DyBM) [O & Otsuka (2015)]

• DyBM with LTP

• DyBM with LTP and LTD

• Experiments [O & Otsuka (2015), Dasgupta & O (2017)]

Page 16: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

LTP: Weight is strengthened when the pre-synaptic neuron fires shortly after the post-synaptic neuron

Bi & Poo (1998)Dan & Poo (2006)

Pre-synapticneuron i

xi

Post-synapticneuron j

xj

Synapsewij

Synapse weakened(Long Term Depression)

Page 17: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Interpreting LTD as negative LTP for a connection from post-synaptic neuron to pre-synaptic neuron

Time from pre-spike to post-spike Time from post-spike to pre-spike

Bi & Poo (1998)Dan & Poo (2006)

Page 18: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Pre-synaptic neuron

Post-synaptic neuron

Image courtesy of dream designs at FreeDigitalPhotos.net

Page 19: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines

x[0]x[-1]x[-T]

Time

𝑇 → ∞ Dynamic Boltzmann machineBoltzmann machine for a T-th order Markov model

Historical values Next value

Weight from neuron i at time t to neuron j at time t + 𝛿

𝑤𝑖𝑗𝛿= 𝑢𝑖𝑗 𝜆

𝛿−𝑑𝑖𝑗

𝛿𝑑𝑖𝑗

𝑤𝑖𝑗𝛿= −𝑣𝑖𝑗 𝜇

−𝛿

Page 20: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines

x[0]x[-1]x[-T]

Time

𝑇 → ∞ Dynamic Boltzmann machineBoltzmann machine for a T-th order Markov model

Historical values Next value

Weight from neuron i at time t to neuron j at time t + 𝛿

𝑤𝑖𝑗𝛿= 𝑢𝑖𝑗 𝜆

𝛿−𝑑𝑖𝑗

𝛿𝑑𝑖𝑗

𝑤𝑖𝑗𝛿= −𝑣𝑖𝑗 𝜇

−𝛿

weight from post to pre

Page 21: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Dynamic Boltzmann machine (LTP & LTD)

xi[t] xj

[t]xi[t-1]

Conduction delay, dij

xi[t-2] xi

[t-3]

𝛽𝑖𝑗𝑡−1

𝑠=−𝑑𝑖𝑗+1

−𝟏

𝑥𝑖𝑡+𝑠

𝜇𝑠

Probability for neuron j to fire at time t:

𝑋𝑗𝑡

≡ 𝑃 𝑥𝑗[𝑡]

= 1|𝑥[:𝑡−1] =1

1 + exp 𝑖 −𝑢𝑖𝑗 𝛼𝑖𝑗𝑡−1

+ 𝑣𝑖𝑗 𝛽𝑖𝑗𝑡−1

+ 𝑣𝑗𝑖 𝛾𝑖𝑡−1

Neural eligibility trace:

𝛾𝑖[𝑡]

← 𝜇 𝛾𝑖𝑡−1

+ 𝑥𝑖𝑡

Synaptic eligibility trace:

𝛼𝑖𝑗[𝑡]

← 𝜆 𝛼𝑖𝑗𝑡−1

+ 𝑥𝑖

𝑡−𝑑𝑖𝑗

Page 22: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Learning rule for Dynamic Boltzmann machine (LTP & LTD)

Stochastic gradient for LTD weight:

𝑣𝑖𝑗 ← 𝑣𝑖𝑗 + 𝜂 𝛽𝑖𝑗𝑡−1

𝑋𝑗𝑡

− 𝑥𝑗𝑡

+ 𝜂 𝛾𝑗𝑡−1

𝑋𝑖𝑡

− 𝑥𝑖𝑡

Stochastic gradient for LTP weight:

𝑢𝑖𝑗 ← 𝑢𝑖𝑗 + 𝜂 𝛼𝑖𝑗𝑡−1

𝑥𝑗𝑡− 𝑋𝑗

𝑡

xi[t] xj

[t]xi[t-1]

Conduction delay, dij

xi[t-2] xi

[t-3]

𝛽𝑖𝑗𝑡−1

𝑠=−𝑑𝑖𝑗+1

−𝟏

𝑥𝑖𝑡+𝑠

𝜇𝑠

Neural eligibility trace:

𝛾𝑖[𝑡]

← 𝜇 𝛾𝑖𝑡−1

+ 𝑥𝑖𝑡

Synaptic eligibility trace:

𝛼𝑖𝑗[𝑡]

← 𝜆 𝛼𝑖𝑗𝑡−1

+ 𝑥𝑖

𝑡−𝑑𝑖𝑗

Page 23: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

We can use multiple eligibility traces with varying decay rates for long term memory

xi[t] xj

[t]xi[t-1]

Conduction delay, dij

xi[t-2] xi

[t-3]

𝛽𝑖𝑗ℓ𝑡−1

𝑠=−𝑑𝑖𝑗+1

−𝟏

𝑥𝑖𝑡+𝑠

𝜇ℓ𝑠

ℓ = 1,2,… 𝑘 = 1,2,…

Neural eligibility trace:

𝛾𝑖[𝑡]

← 𝜇 𝛾𝑖𝑡−1

+ 𝑥𝑖𝑡

Synaptic eligibility trace:

𝛼𝑖𝑗[𝑡]

← 𝜆 𝛼𝑖𝑗𝑡−1

+ 𝑥𝑖

𝑡−𝑑𝑖𝑗

Page 24: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

This talk

• Boltzmann machine, Hebb’s rule, STDP

• Dynamic Boltzmann machine [O & Otsuka (2015)]

• Experiments [O & Otsuka (2015), Dasgupta & O (2017)]

Page 25: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Presenting sequences of 7-bit patterns to a dynamic Boltzmann machine

Page 26: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

DyBM as associative memory for sequential patterns [O & Otsuka (2015)]

Giving DyBM a partial sequence as a cue

DyBM completes the remaining sequence

Energy landscape of DyBM evolves depending on the past patterns

Images from www.nature.com/articles/srep14149

Page 27: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

DyBM detecting anomaly [O & Otsuka (2015)]

Negative log-likelihood

Beginning of anomalous sequence

Image from www.nature.com/articles/srep14149

Page 28: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Time

DyBM learning a generative model of human evolution [O & Otsuka (2015)]

20 neurons

Images from www.nature.com/articles/srep14149

Page 29: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

DyBM learning a generative model of Ich bin ein Musikante[O & Otsuka (2015)]

• 12 neurons• Sounds prepared by Shohei Ohsawa

& Yachiko Obara

Images from www.nature.com/articles/srep14149

Page 30: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Comparison between DyBM and LSTM [Dasgupta & O (AAAI-17)]

Model Retail price of gasoline & diesel• 8 dimensions• 20 hidden units

Sunspot number• 1 dimension• 50 hidden units

LSTM 0.067 0.073

DyBM (delay=2) 0.058 0.082

DyBM (delay=3) 0.056 0.077

DyBM (delay=4) 0.060 0.077

Root mean squared error after learning for 20 epochs

DyBM and LSTM have comparable structures

Systematic approaches to tuning DyBM’s hyperparameters[Dasgupta, Yoshizumi, O (ICPR 2016)][O & Dasgupta (IBM R&D J. 2017)]

Page 31: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Comparison between DyBM and LSTM [Dasgupta & O (AAAI-17)]

DyBM

LSTM (Keras library)

Learning Prediction

Comparable accuracy

RMSE = 0.073

RMSE = 0.077

DyBM learns 16 times faster

Time per epoch = 0.7 sec.

Time per epoch = 11.2 sec.

Learning and predicting sunspot numberImages from IBM Research Report RT0975

Page 32: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

Summary: Dynamic Boltzmann machine

Boltzmann machine

Dynamic Boltzmann machine

Hebb’s rule

Spike-timing dependent plasticity

Bi & Poo (1998)Dan & Poo (2006)

xi

xjwij

MLE

MLE Cells that fire together, wire together

Formal limit Refine

Page 33: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

We provide theoretical underpinnings for STDP

Hebb’s rule (’49)

Motivated artificial neural networks- Perceptron (’58)

Failure

1950 1960 1970 1980 1990

Theoretical underpinnings- Hopfield network (’82)- Boltzmann machine (’83)

2000 2010

Success- Deep learning

STDP (’90s)

Theoretical underpinnings- Dynamic Boltzmann machine[This talk]

Successful applications

Page 34: Dynamic Boltzmann Machine - IBISMLibisml.org/ibis2016/files/2016/09/IBIS2016-osogami.pdf · Dynamic Boltzmann machine as a limit of a sequence of Boltzmann machines x[-T] x[-1] x[0]

References

• Takayuki Osogami and Makoto Otsuka, “Seven neurons memorizing sequences of alphabetical images via spike-timing dependent plasticity,” Scientific Reports, 5, 14149 (2015). www.nature.com/articles/srep14149

• Takayuki Osogami and Makoto Otsuka, Learning dynamic Boltzmann machines with spike-timing dependent plasticity, IBM Research Report, RT0967 (2015). arxiv.org/abs/1509.08634

• Sakyasingha Dasgupta, Takayuki Yoshizumi, and Takayuki Osogami, “Regularized dynamic Boltzmann machine with delay pruning for unsupervised learning of temporal sequences,” ICPR 2016. arxiv.org/abs/1610.01989

• Takayuki Osogami, “Learning binary or real-valued time-series via spike-timing dependent plasticity,” Computing with Spikes NIPS 2016 Workshop, to appear.

• Sakyasingha Dasgupta and Takayuki Osogami, “Nonlinear dynamic Boltzmann machines for time-series prediction,” AAAI-17. Extended research report available at goo.gl/Vd0wna

• Takayuki Osogami and Sakyasingha Dasgupta, “Learning the values of the hyperparametersof a dynamic Boltzmann machine,” IBM Journal of Research and Development, 61(4/5), 2017, to appear.

Dynamic Boltzmann machine

This research was supported by CREST, JST