Neural Networks Introduction - GitHub Pages
Transcript of Neural Networks Introduction - GitHub Pages
![Page 1: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/1.jpg)
Nakul Gopalan
Georgia Tech
Neural Networks
Introduction
Machine Learning CS 4641
These slides are from Vivek Srikumar, Mahdi Roozbahani and Chao Zhang.
![Page 2: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/2.jpg)
Outline
2
• Perceptron
• Stacking Linear Threshold Units
• Neural Networks
• Expressivity of Neural Networks
• Predicting with Neural Networks
• Backpropogation
![Page 3: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/3.jpg)
Linear Classifiers
These slides are from Vivek Srikumar
![Page 4: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/4.jpg)
Linear Classifiers
These slides are from Vivek Srikumar
![Page 5: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/5.jpg)
Perceptron
These slides are from Vivek Srikumar
![Page 6: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/6.jpg)
Perceptron Algorithm
These slides are from Vivek Srikumar
![Page 7: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/7.jpg)
These slides are from Vivek Srikumar
![Page 8: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/8.jpg)
Outline
8
• Perceptron
• Stacking Linear Threshold Units
• Neural Networks
• Expressivity of Neural Networks
• Predicting with Neural Networks
• Backpropogation
![Page 9: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/9.jpg)
Linear Threshold Unit
9
![Page 10: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/10.jpg)
Features for Linear Threshold Unit
10
![Page 11: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/11.jpg)
Features from Classifiers
11
![Page 12: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/12.jpg)
Features from Classifiers
12
![Page 13: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/13.jpg)
Features from Classifiers
13
![Page 14: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/14.jpg)
Features from Classifiers
14
![Page 15: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/15.jpg)
Features from Classifiers
15
![Page 16: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/16.jpg)
Outline
16
• Perceptron
• Stacking Linear Threshold Units
• Neural Networks
• Expressivity of Neural Networks
• Predicting with Neural Networks
• Backpropogation
![Page 17: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/17.jpg)
Neural Networks
17
![Page 18: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/18.jpg)
Inspiration from Biological Neurons
18
![Page 19: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/19.jpg)
Artificial Neurons
19
![Page 20: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/20.jpg)
Activation Functions
20
![Page 21: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/21.jpg)
Neural Network
21
![Page 22: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/22.jpg)
Neural Network
22
![Page 23: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/23.jpg)
A Brief History of Neural Network
23
![Page 24: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/24.jpg)
Outline
24
• Perceptron
• Stacking Linear Threshold Units
• Neural Networks
• Expressivity of Neural Networks
• Predicting with Neural Networks
• Backpropogation
![Page 25: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/25.jpg)
A Single Neuron with Threshold Activation
25
![Page 26: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/26.jpg)
Two Layers with Threshold Activation
26Figure from [Shai Shalev-Shwartz and Shai Ben-David, 2014]
![Page 27: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/27.jpg)
Three Layers with Threshold Activation
27Figure from [Shai Shalev-Shwartz and Shai Ben-David, 2014]
![Page 28: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/28.jpg)
NNs are Universal Function Approximators
28
![Page 29: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/29.jpg)
Outline
29
• Perceptron
• Stacking Linear Threshold Units
• Neural Networks
• Expressivity of Neural Networks
• Predicting with Neural Networks
• Backpropogation
![Page 30: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/30.jpg)
Predicting with Neural Networks
30
![Page 31: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/31.jpg)
Predicting with Neural Networks
31
![Page 32: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/32.jpg)
Predicting with Neural Networks
32
![Page 33: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/33.jpg)
Predicting with Neural Nets: The Forward Pass
33
![Page 34: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/34.jpg)
The Forward Pass
34
![Page 35: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/35.jpg)
The Forward Pass
35
![Page 36: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/36.jpg)
The Forward Pass
36
![Page 37: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/37.jpg)
Outline
37
• Perceptron
• Stacking Linear Threshold Units
• Neural Networks
• Expressivity of Neural Networks
• Predicting with Neural Networks
• Backpropogation
![Page 38: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/38.jpg)
Backpropogation
• Smarter chain rule for derivatives
• What is chain rule:
Univariate case:
Multivariate case (Partial derivatives):
These slides are from Roger Grosse
![Page 39: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/39.jpg)
Backpropogation
These slides are from Matt Gromley
![Page 40: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/40.jpg)
Backpropogation
These slides are from Matt Gromley
![Page 41: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/41.jpg)
Backpropogtion
1
2𝑦 − 𝑦∗ 2 𝑦 − 𝑦∗
These slides are from Matt Gromley
![Page 42: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/42.jpg)
Backpropagation
• Backprop is used to train the overwhelming majority of neural
nets today.
Even optimization algorithms much fancier than gradient descent (e.g.
second-order methods) use backprop to compute the gradients.
• Despite its practical success, backprop is believed to be
neurally implausible. No evidence for biological signals
analogous to error derivatives.
All the biologically plausible alternatives we know about learn much more
slowly (on computers). So how on earth does the brain learn?
Slide from Roger Grosse
![Page 43: Neural Networks Introduction - GitHub Pages](https://reader030.fdocuments.us/reader030/viewer/2022012916/61c6e1d686cc7c1a455e0e27/html5/thumbnails/43.jpg)
Take-Home Messages
43
• Stacking Linear Threshold Units
• Neural Networks
• Expressivity of Neural Networks
• Predicting with Neural Networks
• Backprop is chain rule with some book-keeping