Introduction to Computational Vision Training Neural Nets...
Transcript of Introduction to Computational Vision Training Neural Nets...
![Page 1: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/1.jpg)
Introduction to Computational VisionTraining Neural Nets and CNNs
Agastya Kalra1
![Page 2: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/2.jpg)
Outline
• Training
• Convolutional Layers
Agastya Kalra2
![Page 3: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/3.jpg)
SGD Formalized
3
![Page 4: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/4.jpg)
Momentum
• Adds a velocity term•
• Adds a speedup of at most
• Momentum constant usually 0.9, 0.5, 0.99corresponding to a 10x, 2x, 100x increase in max speed
4
![Page 5: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/5.jpg)
Nesterov Momentum
• Applies momentum first then gradient
5 http://cs231n.stanford.edu/slides/2016/winter1516_lecture6.pdf
![Page 6: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/6.jpg)
Setting the learning rate
6 http://cs231n.stanford.edu/slides/2016/winter1516_lecture6.pdf
● Also good to try a setting for 100 iterations and see which is best on validation set
![Page 7: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/7.jpg)
Others
7
• Adagrad• Adaptive Learning Rates
• Adam• Adaptive Learning Rates + Momentum
• RMSProp• Adaptive learning rates with a slightly different decay
![Page 8: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/8.jpg)
How to choose?
• SGD < SGD+Momentum < SGD+Nesterov Momentum• Adam is a good default• RMSProp is good for RNNs, but also good default• SGD + Nesterov momentum is best if you have
time/resources to optimize learning rate• More of an Art
8
![Page 9: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/9.jpg)
Outline
• Training
• Convolutional Layers
Agastya Kalra9
![Page 10: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/10.jpg)
Fully Connected Layer
10
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 11: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/11.jpg)
Fully Connected Layer
11
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 12: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/12.jpg)
Convolutional Layer
12
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 13: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/13.jpg)
Convolutional Layer
13
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 14: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/14.jpg)
Convolutional Layer
14
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 15: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/15.jpg)
Convolutional Layer
15
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 16: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/16.jpg)
Convolutional Layer
16
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 17: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/17.jpg)
Convolutional Layer
17
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 18: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/18.jpg)
Convolutional Layer
18
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 19: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/19.jpg)
Convolutional Layer
19
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 20: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/20.jpg)
Convolutional Layer
20
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 21: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/21.jpg)
Convolutional Layer
21
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 22: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/22.jpg)
Convolutional Layer
22
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 23: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/23.jpg)
Convolutional Layer
23
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 24: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/24.jpg)
Convolutional Layer
24
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 25: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/25.jpg)
Convolutional Layer
25
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 26: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/26.jpg)
Convolutional Layer
26
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 27: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/27.jpg)
Convolutional Layer
27
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 28: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/28.jpg)
Convolutional Layer
28
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 29: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/29.jpg)
Convolutional Layer
29
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 30: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/30.jpg)
Convolutional Layer
30
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 31: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/31.jpg)
Convolutional Layer
31
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 32: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/32.jpg)
Convolutional Layer
32
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 33: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/33.jpg)
Convolutional Layer
33
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 34: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/34.jpg)
Convolutional Layer
34
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 35: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/35.jpg)
Convolutional Layer
35
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 36: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/36.jpg)
Convolutional Layer
36
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 37: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/37.jpg)
Convolutional Layer
37
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 38: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/38.jpg)
Convolutional Layer
38
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 39: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/39.jpg)
Convolutional Layer
39
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 40: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/40.jpg)
Convolutional Layer
40
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 41: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/41.jpg)
Convolutional Layer
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
41
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf
![Page 42: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/42.jpg)
Convolutional Layer: Im2Col
In Practice, this is actually implemented at a matrix multiplication:
https://www.mathworks.com/help/images/ref/im2col.htmlThen Backprop is the same as backprop through a matrix multiply.
Note: If you want to tie the params of 2 weights, initialize them the same and sum their gradients at each timestep.
42
![Page 43: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fe4bc19305892360c523d64/html5/thumbnails/43.jpg)
43