Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep...
Transcript of Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep...
![Page 2: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/2.jpg)
![Page 3: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/3.jpg)
René Donner Deep Learning
Overview
3
The (amazing) things Deep Learning can do
How does it work?
How can you start with DL?
![Page 4: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/4.jpg)
René Donner Deep Learning
Roughly …
4
Deep learning finds patternsin data corresponding tohigh-level, abstract concepts
![Page 5: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/5.jpg)
What can it do?
![Page 6: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/6.jpg)
René Donner Deep Learning
What it can be used for
6
Image recognition
Text understanding, translation
Voice recognition
Playing video games
Driving cars
…
![Page 7: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/7.jpg)
René Donner Deep Learning
Image recognition
7
![Page 8: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/8.jpg)
René Donner Deep Learning
Scene labeling
8
http://www.purdue.edu/newsroom/releases/2014/Q1/smartphone-to-become-smarter-with-deep-learning-innovation.html
![Page 9: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/9.jpg)
René Donner Deep Learning
Text recognition
9
http://www.pyimagesearch.com/2014/09/22/getting-started-deep-learning-python/
Large-Scale Deep Learning for Intelligent Computer Systems, Jeff Dean, Google, BayLearn 2015
![Page 10: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/10.jpg)
René Donner Deep Learning
Text understanding
10
2013 Glove: Global Vectors for Word Representation, Jeffrey Pennington, Richard Socher and Christopher D. Manning
![Page 11: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/11.jpg)
René Donner Deep Learning
Word embeddings
11
![Page 12: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/12.jpg)
René Donner Deep Learning
Information extraction / Reasoning
12
MetaMind
![Page 13: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/13.jpg)
René Donner Deep Learning
Some well know research groups
13
Stanford / BaiduAndrew Ng
NYU / FacebookYann LeCun
UToronto / GoogleGeoffrey Hinton
![Page 14: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/14.jpg)
René Donner Deep Learning
NVIDIA
14
Images: NVIDIA website
![Page 15: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/15.jpg)
How does it work?
![Page 16: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/16.jpg)
René Donner Deep Learning
Difference to classic ML
16
http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html
![Page 17: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/17.jpg)
René Donner Deep Learning
Deep learning
17
http://theanalyticsstore.ie/deep-learning/
![Page 18: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/18.jpg)
René Donner Deep Learning
Visualization
18
1. Layer
higher Layers
Emergence of Object-Selective Features in Unsupervised Feature Learning, Adam Coates, NIPS 2012
![Page 19: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/19.jpg)
René Donner Deep Learning 19
http://theanalyticsstore.ie/deep-learning/ http://stats.stackexchange.com/questions/114385/what-is-the-difference-between-convolutional-neural-networks-restricted-boltzma
![Page 20: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/20.jpg)
René Donner Deep Learning 20
https://medium.com/machine-learning-world/how-to-debug-neural-networks-manual-dc2a200f10f2
![Page 21: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/21.jpg)
René Donner Deep Learning
Optimization
21
Stochastic gradient descent
Automatic differentiation
blog.datumbox.com
![Page 22: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/22.jpg)
René Donner Deep Learning
Local minima
22
Less problematic than thought - saddle points
https://ganguli-gang.stanford.edu/figures/14.Saddlepoint.jpg
![Page 23: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/23.jpg)
René Donner Deep Learning
Deep learning
23
Low level features of color images
https://www.coursera.org/course/neuralnets
![Page 24: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/24.jpg)
René Donner Deep Learning
Deep learning
24
http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf
![Page 25: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/25.jpg)
René Donner Deep Learning
Inception topologies
25
ImageNet Classification with Deep Convolutional Neural Networks", Alex Krizhevsky
“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015
![Page 26: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/26.jpg)
René Donner Deep Learning
Network Aims
26
“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015
Classification accuracy
Inference speed (e.g. for video)
Size (mobile devices)
Energy per prediction (battery)
![Page 27: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/27.jpg)
René Donner Deep Learning
Model Zoos
27
“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015
Readily trained networks
Transfer learning - adapt to your task
ONNX exchange format
![Page 28: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/28.jpg)
René Donner Deep Learning
MNIST - Code Demo
28
![Page 29: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/29.jpg)
René Donner Deep Learning
MNIST
29
http://deeplearning4j.org/rbm-mnist-tutorial.html
![Page 30: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/30.jpg)
René Donner Deep Learning
Deep learning - why does it work?
30
Can cope with huge amounts of data
Learns small invariances
Overcomplete, sparse, representations
Learn Embedding
Lots of data
Recent advance: it is actually computable!
![Page 31: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/31.jpg)
René Donner Deep Learning
Deep learning - pros
31
Not-domain specific
Supervised / Semi-supervised / Unsupervised
Classification / regression in last layer
Simple math
Hip
![Page 32: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/32.jpg)
René Donner Deep Learning
Deep learning - cons
32
Lots of meta-parameters
Needs a lot of data
Very compute intensive
Hip
![Page 33: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/33.jpg)
Getting started with DL
![Page 34: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/34.jpg)
René Donner Deep Learning
Frameworks
34
Many different DL toolboxes
Efficiency important (GPU)
Attention to numerical issues
![Page 35: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/35.jpg)
René Donner Deep Learning
Frameworks
35
Caffehttp://caffe.berkeleyvision.org/Plain text filesFastest CNN, GPU
Kerashttps://github.com/fchollet/kerasPython, on top of Theano
TensorFlowhttp://tensorflow.org/Python, by Google
MXNethttps://github.com/dmlc/mxnetPython, R, Julia
Slid
e fro
m c
affe
tuto
rial
![Page 36: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/36.jpg)
René Donner Deep Learning
Tensorflow
36
General gradient descent library
![Page 37: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/37.jpg)
René Donner Deep Learning
Tutorials
37
Stanford tutorialhttps://deeplearning.stanford.edu/wiki/index.php/UFLDL_TutorialMatlab code snippets
videolectures.nethttp://videolectures.net/deeplearning2015_montreal/
courserahttps://www.coursera.org/course/neuralnets
![Page 38: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/38.jpg)
René Donner Deep Learning
Practical hints
38
Bengio ArxivPractical Recommendations for Gradient-Based Training of Deep Architectureshttp://arxiv.org/abs/1206.5533http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html
Kaggle http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challengehttp://benanne.github.io/2014/04/05/galaxy-zoo.html
Relevant conferences NIPS (https://sites.google.com/site/deeplearningworkshopnips2013/accepted-papers)CVPR, ICMLMany interesting papers on arxiv.org
![Page 39: Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextflow.com René Donner Deep Learning – an Overview](https://reader031.fdocuments.us/reader031/viewer/2022022512/5ae480767f8b9a90138f0d62/html5/thumbnails/39.jpg)
René Donner Deep Learning
Current research topics
39
Parallelization
What is deep learning, actually?
Alternative, faster, simpler methods
Multi-domain, transfer learning
Generative Adversarial Networks (GANs)