Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural...
Transcript of Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural...
![Page 1: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/1.jpg)
Introduction to Sequence Modeling
“I’m glad that I’m Turing Complete now”
Xinyu ZhouMegvii (Face++) [email protected]
Apr. 2019
![Page 2: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/2.jpg)
Raise your hand and ask,whenever you have questions...
![Page 3: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/3.jpg)
Outline● RNN Basics● Classical RNN Architectures
○ LSTM
● RNN Variants○ RNN with Attention○ RNN with External Memory
■ Neural Turing Machine
● Attention is All You Need (Transformers)● Applications
○ A market of RNNs
![Page 4: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/4.jpg)
Why Sequence Modeling?
![Page 5: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/5.jpg)
Feedforward Neural Networks● Feedforward neural networks can fit any bounded continuous (compact)
function● This is called Universal approximation theorem
https://en.wikipedia.org/wiki/Universal_approximation_theoremCybenko, George. "Approximation by superpositions of a sigmoidal function." Mathematics of Control, Signals, and Systems (MCSS) 2.4 (1989): 303-314.
![Page 6: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/6.jpg)
Bounded Continuous Function is NOT ENOUGH!
How to solve Travelling Salesman Problem?
![Page 7: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/7.jpg)
Bounded Continuous Function is NOT ENOUGH!
How to solve Travelling Salesman Problem?
We Need to be Turing Complete
![Page 8: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/8.jpg)
Sequence ModelingTuring Completeness requires Sequence processing
Array (List) Lisp (List Processing) Programming Language
“Since inception, Lisp was closely connected with the artificial intelligence research community ...” -- Wikipedia
![Page 9: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/9.jpg)
RNN is Turing Complete
Siegelmann, Hava T., and Eduardo D. Sontag. "On the computational power of neural nets." Journal of computer and system sciences 50.1 (1995): 132-150.
![Page 10: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/10.jpg)
Sequence Modeling● How to take a variable length sequence as input?● How to predict a variable length sequence as output?
Model
Sequence Sequence
![Page 11: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/11.jpg)
RNN BasicsThe Appetizer
![Page 12: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/12.jpg)
RNN (8 yuan) > CNN (5 yuan)
![Page 13: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/13.jpg)
RNN DiagramA lonely feedforward cell
![Page 14: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/14.jpg)
RNN DiagramGrows … with more inputs and outputs
![Page 15: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/15.jpg)
RNN Diagram… here comes a brother
(x_1, x_2) comprises a length-2 sequence
![Page 16: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/16.jpg)
RNN Diagram… with shared (tied) weights
x_i: inputsy_i: outputsW: all the sameh_i: internal states that passedalongF: a “pure” function
![Page 17: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/17.jpg)
RNN Diagram… with shared (tied) weights
A simple implementation of F
![Page 18: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/18.jpg)
Categorize RNNs by input/output types
![Page 19: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/19.jpg)
Categorize RNNs by input/output typesMany-to-many
![Page 20: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/20.jpg)
Categorize RNNs by input/output typesMany-to-one
![Page 21: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/21.jpg)
Categorize RNNs by input/output typesOne-to-Many
![Page 22: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/22.jpg)
Categorize RNNs by input/output typesMany-to-Many: Many-to-One + One-to-Many
![Page 23: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/23.jpg)
Many-to-Many ExampleLanguage Model
● Predict next word givenprevious words
● “h” → “he” → “hel” → “hell” → “hello”
![Page 24: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/24.jpg)
Language Modeling● Tell story●● “Heeeeeel”● ⇒ “Heeeloolllell”● ⇒ “Hellooo”● ⇒ “Hello”
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture10.pdf
![Page 25: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/25.jpg)
Language Modeling● Write (nonsense) book in latex
\begin{proof}We may assume that $\mathcal{I}$ is an abelian sheaf on $\mathcal{C}$.\item Given a morphism $\Delta : \mathcal{F} \to \mathcal{I}$is an injective and let $\mathfrak q$ be an abelian sheaf on $X$.Let $\mathcal{F}$ be a fibered complex. Let $\mathcal{F}$ be a category.\begin{enumerate}\item \hyperref[setain-construction-phantom]{Lemma}\label{lemma-characterize-quasi-finite}Let $\mathcal{F}$ be an abelian quasi-coherent sheaf on $\mathcal{C}$.Let $\mathcal{F}$ be a coherent $\mathcal{O}_X$-module. Then$\mathcal{F}$ is an abelian catenary over $\mathcal{C}$.\item The following are equivalent\begin{enumerate}\item $\mathcal{F}$ is an $\mathcal{O}_X$-module.\end{lemma}
![Page 26: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/26.jpg)
Language Modeling● Write (nonsense) book in latex
![Page 27: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/27.jpg)
Many-to-One ExampleSentiment analysis
● “RNNs are awesome!” ⇒ ● “The course project is too hard for me.” ⇒
![Page 28: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/28.jpg)
Many-to-One ExampleSentiment analysis
● “RNNs are awesome!” ⇒ ● “The course project is too hard for me.” ⇒
![Page 29: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/29.jpg)
Many-to-One + One-to-ManyNeural Machine Translation
![Page 30: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/30.jpg)
Many-to-One + One-to-ManyNeural Machine Translation
![Page 31: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/31.jpg)
Many-to-One + One-to-ManyNeural Machine Translation
![Page 32: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/32.jpg)
Many-to-One + One-to-ManyNeural Machine Translation
Encoder
Decoder
![Page 33: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/33.jpg)
Vanishing/Exploding Gradient Problem
“Grow longer! Grow longer!”
![Page 34: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/34.jpg)
Training RNN● “Backpropagation Through Time”
○ Truncated BPTT
● The chain rule of differentiation○ Just Backpropagation
![Page 35: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/35.jpg)
Vanishing/Exploding Gradient Problem● Consider a linear recurrent net with zero inputs●
Bengio, Yoshua, Patrice Simard, and Paolo Frasconi. "Learning long-term dependencies with gradient descent is difficult." IEEE transactions on neural networks 5.2 (1994): 157-166. https://en.wikipedia.org/wiki/Power_iterationhttp://www.cs.cornell.edu/~bindel/class/cs6210-f09/lec26.pdf
![Page 36: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/36.jpg)
Vanishing/Exploding Gradient Problem● Consider a linear recurrent net with zero inputs●●● Singular value of W < 1 ⇒ only if gradient Vanishes● Singular value of W > 1 ⇐ if gradient Explodes
Bengio, Yoshua, Patrice Simard, and Paolo Frasconi. "Learning long-term dependencies with gradient descent is difficult." IEEE transactions on neural networks 5.2 (1994): 157-166. https://en.wikipedia.org/wiki/Power_iterationhttp://www.cs.cornell.edu/~bindel/class/cs6210-f09/lec26.pdf
![Page 37: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/37.jpg)
Vanishing/Exploding Gradient Problem● Consider a linear recurrent net with zero inputs●●● “It is sufficient for the largest eigenvalue λ_1 of the recurrent weight matrix to
be smaller than 1 for long term components to vanish (as t → ∞) and necessary for it to be larger than 1 for gradients to explode.”
Bengio, Yoshua, Patrice Simard, and Paolo Frasconi. "Learning long-term dependencies with gradient descent is difficult." IEEE transactions on neural networks 5.2 (1994): 157-166. https://en.wikipedia.org/wiki/Power_iterationhttp://www.cs.cornell.edu/~bindel/class/cs6210-f09/lec26.pdf
Details are here
![Page 38: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/38.jpg)
Empirical Feasible Length of RNN
RNN
100
![Page 39: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/39.jpg)
Long short-term memory (LSTM) come to the rescue
Vanilla RNN
LSTM
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture10.pdf
![Page 40: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/40.jpg)
Why LSTM works● i: input gate● f: forget gate● o: output gate● g: temporary variable● c: memory cell●● Key observation:
○ If f == 1 (remember past memories), then■ C_t
○ Looks like a ResNet!■
http://people.idsia.ch/~juergen/lstm/sld017.htm
![Page 41: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/41.jpg)
LSTM vs Weight Sharing ResNet
● Difference○ Never forgets○ No intermediate inputs
Cell
vs
![Page 42: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/42.jpg)
Empirical Feasible Length of LSTM
RNN
100LSTM
500
![Page 43: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/43.jpg)
GRU● Similar to LSTM● Let information flow without a
separate memory cell●● Consider
Chung, Junyoung, et al. "Empirical evaluation of gated recurrent neural networks on sequence modeling." arXiv preprint arXiv:1412.3555 (2014).
![Page 44: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/44.jpg)
Empirical Feasible Length of GRU
RNN
100LSTM
500GRU
784
![Page 45: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/45.jpg)
IndRNN● RNN: ● IndRNN:
Li, Shuai, et al. "Independently recurrent neural network (indrnn): Building a longer and deeper rnn." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
Neurons in the same layer are INDEPENDENT!
Interneuron dependence is achieved by STACKING more layers of IndRNN
![Page 46: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/46.jpg)
Empirical Feasible Length of IndRNN
RNN
100LSTM
500GRU
784IndRNN
5,000
![Page 47: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/47.jpg)
Auxiliary Losses● No gradient? Create one!
Recall past回顾过去
Predict future展望未来
![Page 48: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/48.jpg)
Empirical Feasible Length of IndRNN
RNN
100LSTM
500GRU
784IndRNN
5,000Auxiliary Losses
16,000
![Page 49: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/49.jpg)
Summary
Method Feasible Length
RNN < 100
LSTM 500
GRU 784
IndRNN 5,000
Auxiliary Losses 16,000
![Page 50: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/50.jpg)
Simple RNN Extensions“I am a man of value”
![Page 51: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/51.jpg)
Bidirectional RNN (BDRNN)● RNN can go either way● “Peak into the future”● Truncated version used in speech recognition
https://github.com/huseinzol05/Generate-Music-Bidirectional-RNN
![Page 52: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/52.jpg)
2D-RNN: Pixel-RNN● Pixel-RNN● Each pixel depends on its top and left neighbor
Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu. "Pixel recurrent neural networks." arXiv preprint arXiv:1601.06759 (2016).
![Page 53: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/53.jpg)
Pixel-RNN
Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu. "Pixel recurrent neural networks." arXiv preprint arXiv:1601.06759 (2016).
![Page 54: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/54.jpg)
Pixel-RNN Application● Segmentation
Visin, Francesco, et al. "Reseg: A recurrent neural network-based model for semantic segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2016.
![Page 55: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/55.jpg)
Deep RNN● Stack more of them
○ Pros■ More representational power
○ Cons■ Harder to train
● ⇒ Need residual connections along depth
![Page 56: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/56.jpg)
RNN Basics Summary● The evolution of RNN from Feedforward NN● Recurrence as unrolled computation graph● Vanishing/Exploding gradient problem
○ LSTM and variants○ and the relation to ResNet
● Extensions○ BDRNN○ 2DRNN○ Deep-RNN
![Page 57: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/57.jpg)
Interpretation of RNN
![Page 58: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/58.jpg)
Interpreting Gates in LSTM●
Reference:http://karpathy.github.io/2015/05/21/rnn-effectiveness/
![Page 59: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/59.jpg)
Interpreting Gates in LSTM
Reference:http://karpathy.github.io/2015/05/21/rnn-effectiveness/
![Page 60: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/60.jpg)
Turn an RNN to a DFA● DFA: Deterministic finite automaton
○ E.g.: Regular Expression
Weiss, Gail, Yoav Goldberg, and Eran Yahav. "Extracting automata from recurrent neural networks using queries and counterexamples." arXiv preprint arXiv:1711.09576 (2017).
![Page 61: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/61.jpg)
RNN with Attention
![Page 62: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/62.jpg)
Copy a sequence
Input
Output
Solution in Python
Can neural network learn this program purely from data?
![Page 63: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/63.jpg)
What is Attention?● Differentiate entities by its importance
○ spatial attention is related to location○ temporal attention is related to causality
https://distill.pub/2016/augmented-rnns
![Page 64: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/64.jpg)
Attention over Input Sequence● Neural Machine Translation (NMT)
Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv:1409.0473 (2014).
![Page 65: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/65.jpg)
Neural Machine Translation (NMT)● Attention over input sequence● There’re words in two languages that
share the same meaning.● Attention ⇒ Alignment
○ Differentiable, allowing end-to-end training
Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv:1409.0473 (2014).
![Page 66: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/66.jpg)
![Page 67: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/67.jpg)
z
![Page 68: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/68.jpg)
z
![Page 69: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/69.jpg)
z
![Page 70: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/70.jpg)
https://distill.pub/2016/augmented-rnns
![Page 71: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/71.jpg)
https://distill.pub/2016/augmented-rnns
![Page 72: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/72.jpg)
https://distill.pub/2016/augmented-rnns
![Page 73: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/73.jpg)
https://distill.pub/2016/augmented-rnns
![Page 74: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/74.jpg)
https://distill.pub/2016/augmented-rnns
![Page 75: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/75.jpg)
https://distill.pub/2016/augmented-rnns
![Page 76: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/76.jpg)
https://distill.pub/2016/augmented-rnns
![Page 77: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/77.jpg)
https://distill.pub/2016/augmented-rnns
![Page 78: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/78.jpg)
Image Attention: Image Captioning●
Xu, Kelvin, et al. "Show, attend and tell: Neural image caption generation with visual attention." International Conference on Machine Learning. 2015.
![Page 79: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/79.jpg)
input cat
![Page 80: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/80.jpg)
input catfeature cat
![Page 81: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/81.jpg)
input catfeature cat
![Page 82: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/82.jpg)
input catfeature cat
![Page 83: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/83.jpg)
input catfeature cat
![Page 84: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/84.jpg)
input catfeature cat
![Page 85: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/85.jpg)
input catfeature cat
![Page 86: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/86.jpg)
Image Attention: Image Captioning
![Page 87: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/87.jpg)
Image Attention: Image Captioning
![Page 88: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/88.jpg)
Text Recognition● Implicit language model
![Page 89: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/89.jpg)
Text Recognition● Implicit language model
![Page 90: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/90.jpg)
Soft Attention RNN for OCR
CNN
金 口
Column FC金口香牛肉面
金口香牛肉面
Loss1
Loss2
Attention
![Page 91: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/91.jpg)
RNN with External Memory“I Look Like a Computer”
![Page 92: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/92.jpg)
Copy a sequence
Input
Output
![Page 93: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/93.jpg)
Copy a sequence
Input
Output
Solution in Python
![Page 94: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/94.jpg)
Traditional Machine Learning● √ Elementary Operations● √* Logic flow control
○ Decision tree
● × External Memory○ As opposed to internal memory (hidden states)
Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014).
![Page 95: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/95.jpg)
Traditional Machine Learning● √ Elementary Operations● √* Logic flow control● × External Memory
Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014).
![Page 96: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/96.jpg)
Neural Turing Machines (NTM)● NTM is a neural networks with
a working memory● It reads and write multiple times
at each step● Fully differentiable and can be
trained end-to-end
Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014).
An NTM “Cell”
![Page 97: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/97.jpg)
Neural Turing Machines (NTM)● Memory
○ Sdfsdf
http://llcao.net/cu-deeplearning15/presentation/NeuralTuringMachines.pdf
n
m
![Page 98: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/98.jpg)
Neural Turing Machines (NTM)● Read
● Hard indexing ⇒ Soft Indexing○ A distribution of index○ “Attention”
![Page 99: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/99.jpg)
Neural Turing Machines (NTM)● Read
● Hard indexing ⇒ Soft Indexing○ A distribution of index○ “Attention”
MemoryLocations
![Page 100: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/100.jpg)
Neural Turing Machines (NTM)● Read
● Hard indexing ⇒ Soft Indexing○ A distribution of index○ “Attention”
MemoryLocations
![Page 101: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/101.jpg)
Neural Turing Machines (NTM)● Write
○ Write = erase + add
erase
add
![Page 102: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/102.jpg)
Neural Turing Machines (NTM)● Write
○ Write = erase + add
erase
add
![Page 103: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/103.jpg)
Neural Turing Machines (NTM)● Addressing
![Page 104: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/104.jpg)
Neural Turing Machines (NTM)● Addressing● 1. Focusing by Content
● Cosine Similarity
![Page 105: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/105.jpg)
Neural Turing Machines (NTM)● Addressing● 1. Focusing by Content
● Cosine Similarity
![Page 106: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/106.jpg)
Neural Turing Machines (NTM)● 1. Focusing by Content● 2. Interpolate with previous step
![Page 107: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/107.jpg)
Neural Turing Machines (NTM)● 1. Focusing by Content● 2. Interpolate with previous step
![Page 108: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/108.jpg)
Neural Turing Machines (NTM)● 1. Focusing by Content● 2. Interpolate with previous step● 3. Convolutional Shift
![Page 109: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/109.jpg)
Neural Turing Machines (NTM)● 1. Focusing by Content● 2. Interpolate with previous step● 3. Convolutional Shift
![Page 110: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/110.jpg)
Neural Turing Machines (NTM)● 1. Focusing by Content● 2. Interpolate with previous step● 3. Convolutional Shift
![Page 111: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/111.jpg)
Neural Turing Machines (NTM)● 1. Focusing by Content● 2. Interpolate with previous step● 3. Convolutional Shift● 4. Shapening
![Page 112: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/112.jpg)
Neural Turing Machines (NTM)● 1. Focusing by Content● 2. Interpolate with previous step● 3. Convolutional Shift● 4. Shapening
![Page 113: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/113.jpg)
Neural Turing Machines (NTM)● Addressing
One Head
![Page 114: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/114.jpg)
Neural Turing Machines (NTM)● Addressing
One Head
![Page 115: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/115.jpg)
Neural Turing Machines (NTM)● Controller
○ Feedforward○ LSTM
● Take input● Predict all red-circled variables ● Even if a feedforward controller is
used, NTM is an RNN
![Page 116: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/116.jpg)
NTM: Copy Task
NTM
![Page 117: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/117.jpg)
NTM: Copy Task
LSTM
![Page 118: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/118.jpg)
NTM: Copy Task Comparison
NTM
LSTM
![Page 119: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/119.jpg)
Neural Turing Machines (NTM)● Copy Task● Memory heads
loc_writeloc_read
![Page 120: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/120.jpg)
Neural Turing Machines (NTM)● Repeated Copy Task● Memory heads● White cells are positions of
memory heads
![Page 121: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/121.jpg)
Neural Turing Machines (NTM)● Priority Sort
![Page 122: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/122.jpg)
Misc● More networks with memories
○ Memory networks○ Differentiable Neural Computer (DNC)
● Adaptive Computing Time (ACT)● Using different weights for each step
○ HyperNetworks
● Skip-RNN○
![Page 123: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/123.jpg)
CNN forSequence Modeling
![Page 124: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/124.jpg)
RNN: The Good, Bad and Ugly● Good
○ Turing Complete, strong modeling ability
● Bad○ Dependencies between temporal connections make computation slow
■ CNNs are resurging now to predict sequence■ WaveNet
● Ugly○ Generally hard to train ○ In practice: the memorization limit of LSTM is a couple of hundreds of steps.○ The above two fight
![Page 125: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/125.jpg)
RNN’s Rival: WaveNet● Causal Dilated Convolution
Oord, Aaron van den, et al. "Wavenet: A generative model for raw audio." arXiv preprint arXiv:1609.03499 (2016).
![Page 126: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/126.jpg)
Neural GPU● All input is feeded at every step● “横行霸道的 RNN”
Kaiser, Łukasz, and Ilya Sutskever. "Neural gpus learn algorithms." arXiv preprint arXiv:1511.08228 (2015).
Seq
uenc
e
Step
![Page 127: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/127.jpg)
CNN for Sequence Modeling
Pros● Parallelism● Flexible receptive field size● Stable gradients● Low memory requirement for
training. ● Variable length inputs
Cons● Data storage during evaluation● Potential parameter change for a
transfer of domain
Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." arXiv preprint arXiv:1803.01271 (2018).
![Page 128: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/128.jpg)
Attention is All You NeedLook father, look clearer
![Page 129: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/129.jpg)
Transformer● A Neural GPU that changes Convolution to Attention
○ The order of words in the sequence are ignored● Multi-head attention
○ One attention is not enough; Pay more attention
![Page 130: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/130.jpg)
Transformer● Position Encoding
○ Bring back order
Reference: http://jalammar.github.io/illustrated-transformer/
![Page 131: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/131.jpg)
Transformer● Position Encoding combined
![Page 132: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/132.jpg)
Transformer● Trained on BooksCorpus (800M words)
![Page 133: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/133.jpg)
![Page 134: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/134.jpg)
BERT● BERT: Bidirectional Encoder Representations from Transformers.
![Page 135: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/135.jpg)
BERT: Pretraining Task● Starting with transformer, trained with two tasks
○ Predict randomly masked words■ Mask: my dog is hairy → my dog is [MASK]■ Predict: my dog is [MASK] → my dog is hairy
○ Predict whether one sentence follows another■
![Page 136: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/136.jpg)
BERT: Pretraining Dataset● BooksCorpus (800M)● Wikipedia (2,500M)
![Page 137: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/137.jpg)
BERT: Finetune● Small modification
○ Task-specific output
![Page 138: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/138.jpg)
GPT2: Pretraining● Language Modeling● Dataset: WebText
○ 45 million web pages○ 40GB○ Carefully selected○ Focus on diversity of the corpus
Recall ⇒
![Page 139: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/139.jpg)
GPT2: Finetune● Finetune? No finetune!● Zero-shot Test
○ Performing language modeling task○ The “post-process” the results of multiple language modeling generating output and produce
the final output
● Tasks○ Reading Comprehension○ Summarization○ Translation○ Question Answering
● “Achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting”
![Page 140: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/140.jpg)
GPT2: Example
![Page 141: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/141.jpg)
GPT2: GenerationInput: “In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.”
Output: “The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science. Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez.”
![Page 142: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/142.jpg)
GPT2: The Ethics● OpenAI is not releasing (till now) because they are afraid of the abuse of this
model.●
“Due to our concerns about malicious applications of the technology, we are not releasing the trained model. ”
Reference: https://openai.com/blog/better-language-models/
![Page 143: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/143.jpg)
Summary● Transformer = Neural GPU + input/position dependent kernel + full receptive
field● BERT = Transformer + predictive pretraining● GPT2 = Transformer + language model pretraining + diversified dataset
![Page 144: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/144.jpg)
Summary
Vaswani, Ashish, et al. "Attention Is All You Need." arXiv preprint arXiv:1706.03762 (2017).https://research.googleblog.com/2017/08/transformer-novel-neural-network.htmlhttps://courses.cs.ut.ee/MTAT.03.292/2017_fall/uploads/Main/Attention%20is%20All%20you%20need.pdf
![Page 145: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/145.jpg)
More Applications
![Page 146: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/146.jpg)
RNN without a sequence input● Left
○ learns to read out house numbers from left to right
● Right○ a recurrent network generates
images of digits by learning to sequentially add color to a canvas
Ba, Jimmy, Volodymyr Mnih, and Koray Kavukcuoglu. "Multiple object recognition with visual attention." arXiv preprint arXiv:1412.7755 (2014).Gregor, Karol, et al. "DRAW: A recurrent neural network for image generation." arXiv preprint arXiv:1502.04623 (2015).
![Page 147: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/147.jpg)
Generalizing Recurrence● What is recurrence
○ A computation unit with shared parameter occurs at multiple places in the computation graph■ Convolution will do too
○ … with additional states passing among them■ That’s recurrence
● “Recursive”
![Page 148: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/148.jpg)
![Page 149: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/149.jpg)
![Page 150: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/150.jpg)
![Page 151: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/151.jpg)
![Page 152: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/152.jpg)
Recursive Neural Network● Apply when there’s tree structure in data
○ For natural language use The Standford Parserto build the syntax tree given a sentence
http://cs224d.stanford.edu/lectures/CS224d-Lecture10.pdfhttps://nlp.stanford.edu/software/lex-parser.shtml
![Page 153: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/153.jpg)
Recursive Neural Network● Bottom-up aggregation of
information○ Sentiment Analysis
Socher, Richard, et al. "Recursive deep models for semantic compositionality over a sentiment treebank." Proceedings of the 2013 conference on empirical methods in natural language processing. 2013.
![Page 154: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/154.jpg)
Recursive Neural Network● As a lookup table
Andrychowicz, Marcin, and Karol Kurach. "Learning efficient algorithms with hierarchical attentive memory." arXiv preprint arXiv:1602.03218 (2016).
![Page 155: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/155.jpg)
Speech Recognition● Deep Speech 2
○ Spectrogram○ Convolution○ Deep Bidirectional GRU○ FC○ CTC
Amodei, Dario, et al. "Deep speech 2: End-to-end speech recognition in english and mandarin." International Conference on Machine Learning. 2016. Graves, Alex, et al. "Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks." Proceedings of the 23rd international conference on Machine learning. ACM, 2006.
![Page 156: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/156.jpg)
Generating Sequence● Language modeling
○ Input: “A”○ Output: “A quick brown fox jumps over the lazy dog.”
● Handwriting stroke generation○
https://www.cs.toronto.edu/~graves/handwriting.html
![Page 157: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/157.jpg)
Question Answering1. Mary moved to the bathroom2. John went to the hallway3. Where is Mary? 4. Answer: bathroom
Weston, Jason, Sumit Chopra, and Antoine Bordes. "Memory networks." arXiv preprint arXiv:1410.3916 (2014). Sukhbaatar, Sainbayar, Jason Weston, and Rob Fergus. "End-to-end memory networks." Advances in neural information processing systems. 2015. Andreas, Jacob, et al. "Learning to compose neural networks for question answering." arXiv preprint arXiv:1601.01705 (2016). http://cs.umd.edu/~miyyer/data/deepqa.pdfhttps://research.fb.com/downloads/babi/
![Page 158: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/158.jpg)
Question Answering1. Mary moved to the bathroom2. John went to the hallway3. Where is Mary? 4. Answer: bathroom
Weston, Jason, Sumit Chopra, and Antoine Bordes. "Memory networks." arXiv preprint arXiv:1410.3916 (2014). Sukhbaatar, Sainbayar, Jason Weston, and Rob Fergus. "End-to-end memory networks." Advances in neural information processing systems. 2015. Andreas, Jacob, et al. "Learning to compose neural networks for question answering." arXiv preprint arXiv:1601.01705 (2016). http://cs.umd.edu/~miyyer/data/deepqa.pdfhttps://research.fb.com/downloads/babi/
Memory Networks
![Page 159: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/159.jpg)
Visual Question Answering
Antol, Stanislaw, et al. "Vqa: Visual question answering." Proceedings of the IEEE International Conference on Computer Vision. 2015.
![Page 160: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/160.jpg)
Visual Question Answering● Reason the relations among
Objects in image●● “What size is the cylinder that is
left of the brown metal thing that is left of the big sphere”
●● Dataset
○ CLEVR
https://distill.pub/2016/augmented-rnns/http://cs.stanford.edu/people/jcjohns/clevr/
![Page 161: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/161.jpg)
Visual Question Answering● Reason the relations among
Objects in image●● “What size is the cylinder that is
left of the brown metal thing that is left of the big sphere”
●● Dataset
○ CLEVR
https://distill.pub/2016/augmented-rnns/http://cs.stanford.edu/people/jcjohns/clevr/
![Page 162: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/162.jpg)
● Pointer Networks○ Convex Hull○ TSP○ Delaunay triangulation
● Cross-entropy loss on Soft-attention● Application in Vision
○ Object Tracking
MLA Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly. "Pointer networks." Advances in Neural Information Processing Systems. 2015.
Combinatorial Problems
![Page 163: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/163.jpg)
Combinatorial Problems● Pointer Networks
○ Convex Hull○ TSP○ Delaunay triangulation
● Cross-entropy loss on Soft-attention● Application in Vision
○ Object Tracking
MLA Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly. "Pointer networks." Advances in Neural Information Processing Systems. 2015.
![Page 164: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/164.jpg)
Learning to execute● Executing program
Zaremba, Wojciech, and Ilya Sutskever. "Learning to execute." arXiv preprint arXiv:1410.4615 (2014).
![Page 165: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/165.jpg)
Neural Arithmetic Logic Units
Trask, Andrew, et al. "Neural arithmetic logic units." Advances in Neural Information Processing Systems. 2018.
A Shocking FactMost Neural Networks CANNOT generalize on identity function f(x) = x
BecauseNeural Networks does not know basic arithmetics
● Add and Subtract● Multiply and Divide● Exponential and Logarithm
![Page 166: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/166.jpg)
Neural Arithmetic Logic Units
Trask, Andrew, et al. "Neural arithmetic logic units." Advances in Neural Information Processing Systems. 2018.
Saturation Points● Tanh: {-1, 1}● Sigmoid: {0, 1}● ⇒ Tanh * Sigmoid: {-1, 0, 1}
![Page 167: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/167.jpg)
Neural Arithmetic Logic Units
Trask, Andrew, et al. "Neural arithmetic logic units." Advances in Neural Information Processing Systems. 2018.
Number in Extrapolation is 100 times larger than in Interpolation
![Page 168: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/168.jpg)
Mathematical Reasoning● Homework: 76/100● Exam: 50/100● Have no idea of factorization
○ Phew.. RSA encryption is stillsecure.
Saxton, David, et al. "Analysing Mathematical Reasoning Abilities of Neural Models." arXiv preprint arXiv:1904.01557 (2019).
![Page 169: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/169.jpg)
Compress Image● Compete with JPEG
Toderici, George, et al. "Full resolution image compression with recurrent neural networks." arXiv preprint arXiv:1608.05148 (2016).
![Page 170: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/170.jpg)
Model Architecture Search● Use an RNN to produce model architectures
○ Learned using Reinforcement Learning
Zoph, Barret, et al. "Learning transferable architectures for scalable image recognition." arXiv preprint arXiv:1707.07012 (2017).
![Page 171: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/171.jpg)
Model Architecture Search● Use an RNN to produce network architectures
○ Learned using Reinforcement Learning
Zoph, Barret, et al. "Learning transferable architectures for scalable image recognition." arXiv preprint arXiv:1707.07012 (2017).
![Page 172: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/172.jpg)
![Page 173: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/173.jpg)
Meta-Learning
Santoro, Adam, et al. "Meta-learning with memory-augmented neural networks." International conference on machine learning. 2016.
![Page 174: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/174.jpg)
Summary● Sequence Modeling indicates Turing Completeness. ● Progression: RNN → Longer RNN → CNN → Transformer● Capable solving kinds of strange tasks● The author of GPT2 are concerned to release the model because they
consider the model can be abused by malicious applications.○ The beginning of the end? (AI starts to take over the world)
![Page 175: Introduction to Sequence Modeling · 2020. 8. 8. · Neural Turing Machines (NTM) NTM is a neural networks with a working memory It reads and write multiple times at each step Fully](https://reader036.fdocuments.us/reader036/viewer/2022081403/609dd735aaf88e7dc020ed90/html5/thumbnails/175.jpg)