The Story of Elie Wiesel ok_night/author.html peace/Wiesel/homepage.html.
Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David...
Transcript of Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David...
![Page 1: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/1.jpg)
Deep Learning Basics Lecture 5: Convolution
Princeton University COS 495
Instructor: Yingyu Liang
![Page 2: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/2.jpg)
Convolutional neural networks
• Strong empirical application performance
• Convolutional networks: neural networks that use convolution in place of general matrix multiplication in at least one of their layers
for a specific kind of weight matrix 𝑊
ℎ = 𝜎(𝑊𝑇𝑥 + 𝑏)
![Page 3: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/3.jpg)
Convolution
![Page 4: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/4.jpg)
Convolution: math formula
• Given functions 𝑢(𝑡) and 𝑤(𝑡), their convolution is a function 𝑠 𝑡
• Written as
𝑠 𝑡 = ∫ 𝑢 𝑎 𝑤 𝑡 − 𝑎 𝑑𝑎
𝑠 = 𝑢 ∗ 𝑤 or 𝑠 𝑡 = (𝑢 ∗ 𝑤)(𝑡)
![Page 5: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/5.jpg)
Convolution: discrete version
• Given array 𝑢𝑡 and 𝑤𝑡, their convolution is a function 𝑠𝑡
• Written as
• When 𝑢𝑡 or 𝑤𝑡 is not defined, assumed to be 0
𝑠𝑡 =
𝑎=−∞
+∞
𝑢𝑎𝑤𝑡−𝑎
𝑠 = 𝑢 ∗ 𝑤 or 𝑠𝑡 = 𝑢 ∗ 𝑤 𝑡
![Page 6: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/6.jpg)
Illustration 1
a b c d e f
x y z
xb+yc+zd
𝑤 = [z, y, x]𝑢 = [a, b, c, d, e, f]
![Page 7: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/7.jpg)
Illustration 1
a b c d e f
x y z
xc+yd+ze
![Page 8: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/8.jpg)
Illustration 1
a b c d e f
x y z
xd+ye+zf
![Page 9: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/9.jpg)
Illustration 1: boundary case
a b c d e f
x y
xe+yf
![Page 10: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/10.jpg)
Illustration 1 as matrix multiplication
y z
x y z
x y z
x y z
x y z
x y
a
b
c
d
e
f
![Page 11: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/11.jpg)
Illustration 2: two dimensional case
a b c d
e f g h
i j k l
w x
y z
wa + bx + ey + fz
![Page 12: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/12.jpg)
Illustration 2
a b c d
e f g h
i j k l
w x
y z
bw + cx + fy + gz
wa + bx + ey + fz
![Page 13: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/13.jpg)
Illustration 2
a b c d
e f g h
i j k l
w x
y z
bw + cx + fy + gz
wa + bx + ey + fz
Kernel (or filter)
Feature map
Input
![Page 14: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/14.jpg)
Advantage: sparse interaction
Figure from Deep Learning, by Goodfellow, Bengio, and Courville
Fully connected layer, 𝑚 × 𝑛 edges
𝑚 output nodes
𝑛 input nodes
![Page 15: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/15.jpg)
Advantage: sparse interaction
Figure from Deep Learning, by Goodfellow, Bengio, and Courville
Convolutional layer, ≤ 𝑚 × 𝑘 edges
𝑚 output nodes
𝑛 input nodes
𝑘 kernel size
![Page 16: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/16.jpg)
Advantage: sparse interaction
Figure from Deep Learning, by Goodfellow, Bengio, and Courville
Multiple convolutional layers: larger receptive field
![Page 17: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/17.jpg)
Advantage: parameter sharing
Figure from Deep Learning, by Goodfellow, Bengio, and Courville
The same kernel are used repeatedly.E.g., the black edge is the same weightin the kernel.
![Page 18: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/18.jpg)
Advantage: equivariant representations
• Equivariant: transforming the input = transforming the output
• Example: input is an image, transformation is shifting
• Convolution(shift(input)) = shift(Convolution(input))
• Useful when care only about the existence of a pattern, rather than the location
![Page 19: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/19.jpg)
Pooling
![Page 20: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/20.jpg)
Terminology
Figure from Deep Learning, by Goodfellow, Bengio, and Courville
![Page 21: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/21.jpg)
Pooling
• Summarizing the input (i.e., output the max of the input)
Figure from Deep Learning, by Goodfellow, Bengio, and Courville
![Page 22: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/22.jpg)
Advantage
Induce invariance
Figure from Deep Learning, by Goodfellow, Bengio, and Courville
![Page 23: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/23.jpg)
Motivation from neuroscience
• David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary visual cortex), and won Nobel prize for this
• V1 properties• 2D spatial arrangement
• Simple cells: inspire convolution layers
• Complex cells: inspire pooling layers
![Page 24: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/24.jpg)
Variants of convolution and pooling
![Page 25: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/25.jpg)
Variants of convolutional layers
• Multiple dimensional convolution
• Input and kernel can be 3D• E.g., images have (width, height, RBG channels)
• Multiple kernels lead to multiple feature maps (also called channels)
• Mini-batch of images have 4D: (image_id, width, height, RBG channels)
![Page 26: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/26.jpg)
Variants of convolutional layers
• Padding: valid
a b c d e f
x y z
xd+ye+zf
![Page 27: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/27.jpg)
Variants of convolutional layers
• Padding: same
a b c d e f
x y
xe+yf
![Page 28: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/28.jpg)
Variants of convolutional layers
• Stride
Figure from Deep Learning, by Goodfellow, Bengio, and Courville
![Page 29: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/29.jpg)
Variants of convolutional layers
• Others: • Tiled convolution
• Channel specific convolution
• ……
![Page 30: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/30.jpg)
Variants of pooling
• Stride and padding
Figure from Deep Learning, by Goodfellow, Bengio, and Courville
![Page 31: Deep Learning Basics Lecture 5: Convolution · 2016-09-01 · Motivation from neuroscience •David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary](https://reader033.fdocuments.us/reader033/viewer/2022042012/5e72a36fcf216138991695fa/html5/thumbnails/31.jpg)
Variants of pooling
• Max pooling 𝑦 = max{𝑥1, 𝑥2, … , 𝑥𝑘}
• Average pooling 𝑦 = mean{𝑥1, 𝑥2, … , 𝑥𝑘}
• Others like max-out