One-shot Generalization in Deep Generative...
Transcript of One-shot Generalization in Deep Generative...
![Page 1: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/1.jpg)
Reading Group, presented by Minseop Park
One-shot Generalization in Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016
Reference Papers Auto-Encoding Variational Bayes (D.P. Kingma, M. Welling, ICLR 2014) DRAW: A Recurrent Neural Network For Image Generation (K. Gregor et al, ICML 2015) Spatial Transformer Networks (M. Jaderberg et al, NIPS 2015)
![Page 2: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/2.jpg)
One-shot Generalization
Task
Generation of novel variations of a given exemplar
How?
By conditional, sequential generative model
One-shot learning vs. One-shot generalization
![Page 3: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/3.jpg)
DRAW : overview
• DRAW: Deep Recurrent Attentive Writer
• Basic model of sequential generative model
• Sequential VAE + attention
• Idea : Images like MNIST are generated sequentially
• https://www.youtube.com/watch?v=Zt-7MI9eKEo
![Page 4: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/4.jpg)
Variational Autoencoder
• Optimization of Variation Lower Bound
FNN(Φ)
FNN(θ)
Encoder network
Decoder network
Gaussian Parameters
Bernoulli (Gaussian) Parameter
:
:
![Page 5: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/5.jpg)
DRAW Key Features
• Encoder / Decoder Network : LSTM
• Additive Canvas
• Attention ; where to read, where/what to write
![Page 6: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/6.jpg)
DRAW
𝐶↓𝑡 : canvas matrix 𝐶↓𝑇 is used to parameterize 𝑃(𝑥|𝑧) read/write : attention mechanism
![Page 7: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/7.jpg)
DRAW
-Approximate posterior 𝑄𝑧↓𝑡 ℎ↓𝑡↑𝑒𝑛𝑐 =𝑁𝑧↓𝑡 𝜇↓𝑡 , σ↓𝑡 where
-Data distribution 𝑃𝑥 𝑧↓1:𝑇 =𝐵(𝑥|𝝈(𝑐↓𝑇 ))
-Data Generation
![Page 8: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/8.jpg)
Selective Attention Model
• from the A x B input image, to obtain and N x N attention patch
• Horizontal and vertical filterbank 𝐹↓𝑋 (N x A) and 𝐹↓𝑌 (N x B)
![Page 9: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/9.jpg)
Selective Attention Model
• Attention Parameters are obtained from LSTM output at each time step
• Initial patch covers the whole input image
![Page 10: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/10.jpg)
Read / Write Attention
• Read : from the A x B input image, to obtain and N x N attention patch
• Write : from the N x N attention patch, back to A x B input image
![Page 11: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/11.jpg)
DRAW : results
![Page 12: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/12.jpg)
Sequential Generative Model
• Attention model : 2D Gaussian to Spatial Transformer
• Downsize the # of parameters by cutting the connection of canvas to hidden state
• Conditional generative model
![Page 13: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/13.jpg)
Conditional Generative Model
𝑓↓ℎ : LSTM, state transition 𝑓↓𝑐 : Additive Canvas 𝑓↓𝑣 : read attention 𝑓↓𝑤 : write attention (Spatial Transformer)
![Page 14: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/14.jpg)
Spatial Transformer
• Differentiable attention model with affine transformation
• Learn parameters of affine transform
![Page 15: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/15.jpg)
Result comparison
![Page 16: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/16.jpg)
One-shot generalization : 3 tasks
• Task 1 : Unconditional free generation
• Task 2 : Generation of novel variations of a given exemplar
• Task 3 : Generation of representative samples from a novel alphabet
Result for task 1
![Page 17: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/17.jpg)
One-shot generalization : results
![Page 18: One-shot Generalization in Deep Generative Modelmlg.postech.ac.kr/~readinglist/slides/20160919.pdf · Deep Generative Model Danilo J. Rezende, Shakir Mohamed, ICML 2016 Reference](https://reader030.fdocuments.us/reader030/viewer/2022041015/5ec613b67de35c0582412e3b/html5/thumbnails/18.jpg)
Thank you