SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled...
Transcript of SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled...
![Page 1: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/1.jpg)
DEEP LEARNING SERIES (WWC-AI)
Tim ScarfeMachine learning appreciator from the UK!
http://aka.ms/mdml
MACHINE LEARNING @ MICROSOFT
![Page 2: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/2.jpg)
TALK OUTLINE• Deep learning intro
• Distilled concepts of deep learning
• Why are neural networks good at sequence processing?
• What is sequence processing?
• Working with text data
• Recurrent neural networks
• 1d convolutional neural networks
![Page 3: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/3.jpg)
WHAT IS A NEURAL NETWORK?
![Page 4: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/4.jpg)
WHAT IS A DEEP NEURAL NETWORK?
![Page 5: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/5.jpg)
DISTILLED CONCEPTS OF DEEP LEARNING
![Page 6: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/6.jpg)
ENTIRE MACHINE IS TRAINABLE
• The networks have many levels of depth
• Machine learns a hierarchy of representations
• No feature extraction required
Traditional ML
Feature Extractor Trainable Classifier
Deep Learning
Low Level Features Trainable ClassifierMid Level Features High Level Features
Representations are hierarchical and trained automatically
Hand crafted features
![Page 7: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/7.jpg)
UNIVERSAL FUNCTION APPROXIMATORS
• Unlike other shallow ML algorithms; you can map between data domains
SEQUENCES
We learn a spatial transformation between the twoSPEECH
LANGUAGE
IMAGES
[…]
SEQUENCES
SPEECH
LANGUAGE
IMAGES
[…]
![Page 8: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/8.jpg)
NATIVE DATA-DOMAIN FEATURES
Unlike other algorithms, NNs can natively encode useful and obvious relationships in the data domain
• Local spatial dependencies (vision)
• Time dependencies (language, speech)
Recurrent Neural Networks
Convolutional Neural Networks
![Page 9: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/9.jpg)
COMPOSABILITY
• Composability
• Deep Learning research is very applied
• Accessibility
• Software analogy
![Page 10: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/10.jpg)
WHAT IS A SEQUENCE?
![Page 11: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/11.jpg)
WHAT IS SEQUENCE PROCESSING• RNNs
• Timeseries classification
• Anomaly detection in timeseries
• Entity recognition
• Revenue forecasting
• Question + Answer
• 1d Convnets
• Spelling correction
• Document classification
• Machine translation
![Page 12: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/12.jpg)
WHAT IS SEQUENCE PROCESSING• RNNs
• When global order matters
• 1d Convnets
• Speed
• Local temporal dependencies
• You can stack them!
![Page 13: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/13.jpg)
TOKENIZATION
• Words
• Characters
• N-grams
N-Grams example
![Page 14: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/14.jpg)
WORD VECTORS VS WORD EMBEDDINGS?
![Page 15: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/15.jpg)
WORD EMBEDDINGS
• Word2Vec
• GloVe
![Page 16: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/16.jpg)
RECURRENT NEURAL NETWORKS
![Page 17: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/17.jpg)
RECURRENT NEURAL NETWORKS
Image © Francois Chollet
![Page 18: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/18.jpg)
LSTMS(1)
Image © Francois Chollet
![Page 19: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/19.jpg)
LSTMS(2)
![Page 20: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/20.jpg)
LSTMS(3)
Image © Francois Chollet
![Page 21: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/21.jpg)
LSTM VS GRU
LSTM
GRU
![Page 22: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/22.jpg)
BI-DIRECTIONAL LSTMS
![Page 23: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/23.jpg)
1D-CNNS
![Page 24: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/24.jpg)
2DCNNS – SAME CONCEPT
Low Level Features Mid Level Features
High Level Features
![Page 25: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/25.jpg)
STACKING IS COOL
![Page 26: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/26.jpg)
UNIVERSAL MACHINE LEARNING PROCESS
• Define problem
• Define success
• Validation process
• Vectorise/normalize data
• Develop naïve model
• Refine model based on validation performance
![Page 27: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/27.jpg)
DEMOS OF RNNS
![Page 28: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/28.jpg)
https://www.manning.com/books/deep-learning-with-python
![Page 29: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What](https://reader033.fdocuments.us/reader033/viewer/2022053004/5f07ec727e708231d41f7055/html5/thumbnails/29.jpg)
THANK YOU!