Microdecision making in financial services - Greg Lamp @ PAPIs Connect
Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect
-
Upload
papisio -
Category
Technology
-
view
410 -
download
1
Transcript of Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect
![Page 1: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/1.jpg)
Deep Learning
Roberto Paredes ([email protected])PRHLT Research Center
Universitat Politecnica de Valencia
March 2016
![Page 2: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/2.jpg)
Deep Learning Introduction
• Neural networks
• Deep Learning: Stack many layers to build deep models
• Recently, grab the attention of the industry
• Many new applications
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 2
![Page 3: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/3.jpg)
Deep Learning Introduction
http://qz.com/335768/bill-gates-joins-elon-musk-and-stephen-hawking-in-saying-artificial-intelligence-is-scary/
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 3
![Page 4: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/4.jpg)
Deep Learning Introduction
• Key issue: Representational Learning
• Seamless Representation-Classification model
... and make it happen!
... and even make it affordable!
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 4
![Page 5: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/5.jpg)
Deep Learning Introduction
• Neural Network:
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 5
![Page 6: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/6.jpg)
Deep Learning Introduction
• Deep Learning → Bridge the gap between raw representation and categories
http://www.clarifai.com/static/img_ours/cnn.png
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 6
![Page 7: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/7.jpg)
Deep Learning Applications
• Deep Learning:
– Key issue Representational Learning– Some realistic problems require a deep structure to be learned properly
• Applications:
– Image Recognition– Speech Recognition– Natural Language Processing– Machine Translation– ...
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 7
![Page 8: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/8.jpg)
Deep Learning Achievements - Computer Vision
• ImageNet Challenge
http://blogs.nvidia.com/blog/2014/09/18/gpus-imagenet-deep-learning/
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 8
![Page 9: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/9.jpg)
Deep Learning Achievements - Computer Vision and NLP
http://googleresearch.blogspot.com.es/2014/11/a-picture-is-worth-thousand-coherent.html
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 9
![Page 10: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/10.jpg)
Deep Learning Achievements - Computer Vision and NLP
http://googleresearch.blogspot.com.es/2014/11/a-picture-is-worth-thousand-coherent.html
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 10
![Page 11: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/11.jpg)
Deep Learning Achievements - Computer Vision
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional
neural networks. In Advances in neural information processing systems (pp. 1097-1105).
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 11
![Page 12: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/12.jpg)
Deep Learning Achievements - Computer Vision
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 12
![Page 13: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/13.jpg)
Deep Learning Achievements - Computer Vision
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 13
![Page 14: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/14.jpg)
Deep Learning Achievements - Speech Recognition
• Tandem DBN-DNN-HMM
https://www.cs.toronto.edu/~hinton/absps/DNN-2012-proof.pdf
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 14
![Page 15: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/15.jpg)
Deep Learning Achievements - Handwritten Text Recognition
• Bidirectional LSTM. CTC and LM
• ICDAR competition, WER: from 27 (Basic HMM) down to 15 (Combinations)
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 15
![Page 16: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/16.jpg)
Deep Learning (Neural Networks) Approaches
• Neural Network (Multi-Layer Perceptron)
– Deep Neural Network (Multi-Layer Perceptron with more layers)– Word2Vec (NLP) (Multi-Layer Perceptron)– Autoencoders and Denoising Autoencoders (Multi-Layer Perceptron)
• Convolutional Neural Networks
• Long-Short Term Memory LSTM
• Restricted Boltzmann Machines
– Deep Belief Networks– Deep Boltzmann Machines
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 16
![Page 17: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/17.jpg)
Deep Learning (Neural Networks) Approaches (Dates)
• Neural Network (Multi-Layer Perceptron) (1986)
• Convolutional Neural Networks (1989)
• Long-Short Term Memory LSTM (1997)
• Restricted Boltzmann Machines (2006)
– Deep Belief Networks– Deep Boltzmann Maniches
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 17
![Page 18: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/18.jpg)
Deep Learning (Neural Networks) approaches
• Supervised:
– DNN– Convolutional NN– LSTM
Training: Backpropagation
• Unsupervised:
– Stacked Autoencoders
Training: Backpropagation
– Restricted Boltzmann Machines
Contrastive Divergence or Persistent CD
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 18
![Page 19: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/19.jpg)
Deep Learning problems
• Problem when backpropagating errors to first layers
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 19
![Page 20: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/20.jpg)
Deep Learning problems
• Nowadays a DeepNet for computer vision is something like this:
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 20
![Page 21: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/21.jpg)
Deep Learning problems
• Backpropagation , Why now it works with deep structures?
– Tons of data– Sharing weights on very initial layers (CNN)– Improving Generalization:∗ Dropout∗ Dropconnect∗ Denoising Autoencoders
– New activation functions:∗ ReLU∗ MaxOut⇒ piecewise-linear behaviour and constant gradients
– Layer by layer training– Virtual Data with common distortions– Hardware allows to run experiments!! (GPU)
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 21
![Page 22: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/22.jpg)
Deep Learning things to consider
• Supervised training (DNN,CNN):
– Network topology– Sigmoid, tanh, ReLu, softmax, linear, Maxout– Data normalization– Virtual data– Batch size, Epochs– Weights initialization– Learning rate– Momentum rate– Dropout– Dropconnect– ...
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 22
![Page 23: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/23.jpg)
Deep Learning, when?
• When to use Deep Learning:
– There is an big gap between raw representation and categories– Hand-crafted features didn’t work– There are a lot of data for training (or virtual distortions)– Good hardware is available
• When hand-crafted features are good (expert knowledge):
– SVM– Random Forests, ERT– AdaBoost– ...
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 23
![Page 24: Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect](https://reader033.fdocuments.us/reader033/viewer/2022052915/5872f0701a28ab8c718b47c3/html5/thumbnails/24.jpg)
Deep Learning
Thanks for your attention
March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 24