Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize...
Transcript of Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize...
![Page 1: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/1.jpg)
Image Segmentation and Visualization
Xiaolong Wang
![Page 2: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/2.jpg)
This Class
• Naïve FCN model for Image Segmentation
• Transpose Convolution
• Skip Connection, Hypercolumn
• Visualizing Deep Networks
Slides partially from: http://cs231n.stanford.edu/ https://slazebni.cs.illinois.edu/fall20/
![Page 3: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/3.jpg)
Segmentation Problem and FCN
![Page 4: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/4.jpg)
The problem
semantic segmentation instance segmentation
image classification object detection
![Page 5: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/5.jpg)
The problem
![Page 6: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/6.jpg)
Semantic Segmentation
Simply staring one pixel is impossible to do the classification
Let’s put in some context!
![Page 7: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/7.jpg)
Semantic Segmentation
![Page 8: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/8.jpg)
Semantic Segmentation
Time Consuming!
![Page 9: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/9.jpg)
Can we process the whole image at one time?
AlexNet input: 227 x 277 x 3
AlexNet Conv5: 13 x 13 x 128
Output ?13 x 13 x 21
Output is too small!
![Page 10: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/10.jpg)
Can we process the whole image at one time?
AlexNet input: 227 x 277 x 3
AlexNet Conv5: 13 x 13 x 128
Output ?227 x 227 x 21
FC + reshape
Huge number of Parameters for FC: 13 x 13 x 128 x 227 x 227 x 21
![Page 11: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/11.jpg)
Fully Convolutional Network
Convolution at original image resolution has high computation cost.
![Page 12: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/12.jpg)
Fully Convolutional Network
D! × H/4 ×W/4 D! × H/4 ×W/4
D" × H/8 ×W/8
Make the feature map small increases the receptive field
Make the feature map larger again increases the resolution
Transpose Convolution
![Page 13: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/13.jpg)
The FCN paper
Long et al. 2015
![Page 14: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/14.jpg)
1 x 1 convolutions
![Page 15: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/15.jpg)
The FCN paper
![Page 16: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/16.jpg)
The FCN paper
![Page 17: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/17.jpg)
The upsamling
• Upsampling Layer
• Deconvolution Layer
• Transpose Convolution Layer
![Page 18: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/18.jpg)
Transpose Convolution
![Page 19: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/19.jpg)
Recall Convolution3 X 3 convolution with stride 1 and padding 1
![Page 20: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/20.jpg)
Recall Convolution3 X 3 convolution with stride 1 and padding 1
![Page 21: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/21.jpg)
Recall Convolution3 X 3 convolution with stride 2 and padding 1
![Page 22: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/22.jpg)
Recall Convolution3 X 3 convolution with stride 2 and padding 1
![Page 23: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/23.jpg)
Transpose Convolution3 X 3 transpose convolution, stride 2 and padding 1
![Page 24: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/24.jpg)
Transpose Convolution3 X 3 transpose convolution, stride 2 and padding 1
Sum over the overlapping region
![Page 25: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/25.jpg)
1D Transpose Convolution Example
![Page 26: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/26.jpg)
2D ConvolutionRegular convolution (stride 1, pad 0)
𝑥(( 𝑥() 𝑥(* 𝑥(+
𝑥)( 𝑥)) 𝑥)* 𝑥)+
𝑥*( 𝑥*) 𝑥** 𝑥*+
𝑥+( 𝑥+) 𝑥+* 𝑥++
𝑤(( 𝑤() 𝑤(*
𝑤)( 𝑤)) 𝑤)*
𝑤*( 𝑤*) 𝑤**
𝑧(( 𝑧()
𝑧)( 𝑧))
𝑤!! 𝑤!" 𝑤!# 0 𝑤"! 𝑤"" 𝑤"# 0 𝑤#! 𝑤#" 𝑤## 0 0 0 0 00 𝑤!! 𝑤!" 𝑤!# 0 𝑤"! 𝑤"" 𝑤"# 0 𝑤#! 𝑤#" 𝑤## 0 0 0 00 0 0 0 𝑤!! 𝑤!" 𝑤!# 0 𝑤"! 𝑤"" 𝑤"# 0 𝑤#! 𝑤#" 𝑤## 00 0 0 0 0 𝑤!! 𝑤!" 𝑤!# 0 𝑤"! 𝑤"" 𝑤"# 0 𝑤#! 𝑤#" 𝑤##
𝑥##𝑥#!𝑥#"𝑥#$⋮𝑥$$
𝑧##𝑧#!𝑧!#𝑧!!
Matrix-vector form:
∗ =
=
4x4 input, 2x2 output
![Page 27: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/27.jpg)
𝑥!!𝑥!"𝑥!#𝑥!$𝑥"!𝑥""𝑥"#𝑥"$𝑥#!𝑥#"𝑥##𝑥#$𝑥$!𝑥$"𝑥$#𝑥$$
𝑧##𝑧#!𝑧!#𝑧!!
𝑤!! 0 0 0𝑤!" 𝑤!! 0 0𝑤!# 𝑤!" 0 00 𝑤!# 0 0𝑤"! 0 𝑤!! 0𝑤"" 𝑤"! 𝑤!" 𝑤!!𝑤"# 𝑤"" 𝑤!# 𝑤!"0 𝑤"# 0 𝑤!#𝑤#! 0 𝑤"! 0𝑤#" 𝑤#! 𝑤"" 𝑤"!𝑤## 𝑤#" 𝑤"# 𝑤""0 𝑤## 0 𝑤"#0 0 𝑤#! 00 0 𝑤#" 𝑤#!0 0 𝑤## 𝑤#"0 0 0 𝑤##
𝑥!! 𝑥!" 𝑥!# 𝑥!$
𝑥"! 𝑥"" 𝑥"# 𝑥"$
𝑥#! 𝑥#" 𝑥## 𝑥#$
𝑥$! 𝑥$" 𝑥$# 𝑥$$
𝑤!! 𝑤!" 𝑤!#
𝑤"! 𝑤"" 𝑤"#
𝑤#! 𝑤#" 𝑤##
𝑧!! 𝑧!"
𝑧"! 𝑧""
=2x2 input, 4x4 output
=∗%
Not an inverse of the original convolution operation, simply reverses dimension change!
SourceTranspose Convolution
![Page 28: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/28.jpg)
𝑥!!𝑥!"𝑥!#𝑥!$𝑥"!𝑥""𝑥"#𝑥"$𝑥#!𝑥#"𝑥##𝑥#$𝑥$!𝑥$"𝑥$#𝑥$$
𝑧##𝑧#!𝑧!#𝑧!!
𝑤!! 0 0 0𝑤!" 𝑤!! 0 0𝑤!# 𝑤!" 0 00 𝑤!# 0 0𝑤"! 0 𝑤!! 0𝑤"" 𝑤"! 𝑤!" 𝑤!!𝑤"# 𝑤"" 𝑤!# 𝑤!"0 𝑤"# 0 𝑤!#𝑤#! 0 𝑤"! 0𝑤#" 𝑤#! 𝑤"" 𝑤"!𝑤## 𝑤#" 𝑤"# 𝑤""0 𝑤## 0 𝑤"#0 0 𝑤#! 00 0 𝑤#" 𝑤#!0 0 𝑤## 𝑤#"0 0 0 𝑤##
𝑥!! 𝑥!" 𝑥!# 𝑥!$
𝑥"! 𝑥"" 𝑥"# 𝑥"$
𝑥#! 𝑥#" 𝑥## 𝑥#$
𝑥$! 𝑥$" 𝑥$# 𝑥$$
𝑤!! 𝑤!" 𝑤!#
𝑤"! 𝑤"" 𝑤"#
𝑤#! 𝑤#" 𝑤##
𝑧!! 𝑧!"
𝑧"! 𝑧""
𝑥## = 𝑤##𝑧##
=
=∗%
Transpose Convolution
![Page 29: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/29.jpg)
𝑥!!𝑥!"𝑥!#𝑥!$𝑥"!𝑥""𝑥"#𝑥"$𝑥#!𝑥#"𝑥##𝑥#$𝑥$!𝑥$"𝑥$#𝑥$$
𝑧##𝑧#!𝑧!#𝑧!!
𝑤!! 0 0 0𝑤!" 𝑤!! 0 0𝑤!# 𝑤!" 0 00 𝑤!# 0 0𝑤"! 0 𝑤!! 0𝑤"" 𝑤"! 𝑤!" 𝑤!!𝑤"# 𝑤"" 𝑤!# 𝑤!"0 𝑤"# 0 𝑤!#𝑤#! 0 𝑤"! 0𝑤#" 𝑤#! 𝑤"" 𝑤"!𝑤## 𝑤#" 𝑤"# 𝑤""0 𝑤## 0 𝑤"#0 0 𝑤#! 00 0 𝑤#" 𝑤#!0 0 𝑤## 𝑤#"0 0 0 𝑤##
𝑥!! 𝑥!" 𝑥!# 𝑥!$
𝑥"! 𝑥"" 𝑥"# 𝑥"$
𝑥#! 𝑥#" 𝑥## 𝑥#$
𝑥$! 𝑥$" 𝑥$# 𝑥$$
𝑤!! 𝑤!" 𝑤!#
𝑤"! 𝑤"" 𝑤"#
𝑤#! 𝑤#" 𝑤##
𝑧!! 𝑧!"
𝑧"! 𝑧""
𝑥#! = 𝑤#!𝑧## + 𝑤##𝑧#!
=
Convolve input with flipped filter
=∗%
Transpose Convolution
![Page 30: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/30.jpg)
𝑥!!𝑥!"𝑥!#𝑥!$𝑥"!𝑥""𝑥"#𝑥"$𝑥#!𝑥#"𝑥##𝑥#$𝑥$!𝑥$"𝑥$#𝑥$$
𝑧##𝑧#!𝑧!#𝑧!!
𝑤!! 0 0 0𝑤!" 𝑤!! 0 0𝑤!# 𝑤!" 0 00 𝑤!# 0 0𝑤"! 0 𝑤!! 0𝑤"" 𝑤"! 𝑤!" 𝑤!!𝑤"# 𝑤"" 𝑤!# 𝑤!"0 𝑤"# 0 𝑤!#𝑤#! 0 𝑤"! 0𝑤#" 𝑤#! 𝑤"" 𝑤"!𝑤## 𝑤#" 𝑤"# 𝑤""0 𝑤## 0 𝑤"#0 0 𝑤#! 00 0 𝑤#" 𝑤#!0 0 𝑤## 𝑤#"0 0 0 𝑤##
𝑥!! 𝑥!" 𝑥!# 𝑥!$
𝑥"! 𝑥"" 𝑥"# 𝑥"$
𝑥#! 𝑥#" 𝑥## 𝑥#$
𝑥$! 𝑥$" 𝑥$# 𝑥$$
𝑤!! 𝑤!" 𝑤!#
𝑤"! 𝑤"" 𝑤"#
𝑤#! 𝑤#" 𝑤##
𝑧!! 𝑧!"
𝑧"! 𝑧""
𝑥#" = 𝑤#"𝑧## + 𝑤#!𝑧#!
=
=∗%
Convolve input with flipped filter
Transpose Convolution
![Page 31: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/31.jpg)
𝑥!!𝑥!"𝑥!#𝑥!$𝑥"!𝑥""𝑥"#𝑥"$𝑥#!𝑥#"𝑥##𝑥#$𝑥$!𝑥$"𝑥$#𝑥$$
𝑧##𝑧#!𝑧!#𝑧!!
𝑤!! 0 0 0𝑤!" 𝑤!! 0 0𝑤!# 𝑤!" 0 00 𝑤!# 0 0𝑤"! 0 𝑤!! 0𝑤"" 𝑤"! 𝑤!" 𝑤!!𝑤"# 𝑤"" 𝑤!# 𝑤!"0 𝑤"# 0 𝑤!#𝑤#! 0 𝑤"! 0𝑤#" 𝑤#! 𝑤"" 𝑤"!𝑤## 𝑤#" 𝑤"# 𝑤""0 𝑤## 0 𝑤"#0 0 𝑤#! 00 0 𝑤#" 𝑤#!0 0 𝑤## 𝑤#"0 0 0 𝑤##
𝑥!! 𝑥!" 𝑥!# 𝑥!$
𝑥"! 𝑥"" 𝑥"# 𝑥"$
𝑥#! 𝑥#" 𝑥## 𝑥#$
𝑥$! 𝑥$" 𝑥$# 𝑥$$
𝑤!! 𝑤!" 𝑤!#
𝑤"! 𝑤"" 𝑤"#
𝑤#! 𝑤#" 𝑤##
𝑧!! 𝑧!"
𝑧"! 𝑧""
𝑥#$ = 𝑤#"𝑧#!
=
=∗%
Convolve input with flipped filter
Transpose Convolution
![Page 32: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/32.jpg)
𝑥!!𝑥!"𝑥!#𝑥!$𝑥"!𝑥""𝑥"#𝑥"$𝑥#!𝑥#"𝑥##𝑥#$𝑥$!𝑥$"𝑥$#𝑥$$
𝑧##𝑧#!𝑧!#𝑧!!
𝑤!! 0 0 0𝑤!" 𝑤!! 0 0𝑤!# 𝑤!" 0 00 𝑤!# 0 0𝑤"! 0 𝑤!! 0𝑤"" 𝑤"! 𝑤!" 𝑤!!𝑤"# 𝑤"" 𝑤!# 𝑤!"0 𝑤"# 0 𝑤!#𝑤#! 0 𝑤"! 0𝑤#" 𝑤#! 𝑤"" 𝑤"!𝑤## 𝑤#" 𝑤"# 𝑤""0 𝑤## 0 𝑤"#0 0 𝑤#! 00 0 𝑤#" 𝑤#!0 0 𝑤## 𝑤#"0 0 0 𝑤##
𝑥!! 𝑥!" 𝑥!# 𝑥!$
𝑥"! 𝑥"" 𝑥"# 𝑥"$
𝑥#! 𝑥#" 𝑥## 𝑥#$
𝑥$! 𝑥$" 𝑥$# 𝑥$$
𝑤!! 𝑤!" 𝑤!#
𝑤"! 𝑤"" 𝑤"#
𝑤#! 𝑤#" 𝑤##
𝑧!! 𝑧!"
𝑧"! 𝑧""
𝑥!# = 𝑤!#𝑧## + 𝑤##𝑧!#
=
=∗%
Convolve input with flipped filter
Transpose Convolution
![Page 33: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/33.jpg)
𝑥!!𝑥!"𝑥!#𝑥!$𝑥"!𝑥""𝑥"#𝑥"$𝑥#!𝑥#"𝑥##𝑥#$𝑥$!𝑥$"𝑥$#𝑥$$
𝑧##𝑧#!𝑧!#𝑧!!
𝑤!! 0 0 0𝑤!" 𝑤!! 0 0𝑤!# 𝑤!" 0 00 𝑤!# 0 0𝑤"! 0 𝑤!! 0𝑤"" 𝑤"! 𝑤!" 𝑤!!𝑤"# 𝑤"" 𝑤!# 𝑤!"0 𝑤"# 0 𝑤!#𝑤#! 0 𝑤"! 0𝑤#" 𝑤#! 𝑤"" 𝑤"!𝑤## 𝑤#" 𝑤"# 𝑤""0 𝑤## 0 𝑤"#0 0 𝑤#! 00 0 𝑤#" 𝑤#!0 0 𝑤## 𝑤#"0 0 0 𝑤##
𝑥!! 𝑥!" 𝑥!# 𝑥!$
𝑥"! 𝑥"" 𝑥"# 𝑥"$
𝑥#! 𝑥#" 𝑥## 𝑥#$
𝑥$! 𝑥$" 𝑥$# 𝑥$$
𝑤!! 𝑤!" 𝑤!#
𝑤"! 𝑤"" 𝑤"#
𝑤#! 𝑤#" 𝑤##
𝑧!! 𝑧!"
𝑧"! 𝑧""
=
=
𝑥!! = 𝑤!!𝑧## + 𝑤!#𝑧#! + 𝑤#!𝑧!# + 𝑤##𝑧!!
∗%
Convolve input with flipped filter
Transpose Convolution
![Page 34: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/34.jpg)
𝑥!!𝑥!"𝑥!#𝑥!$𝑥"!𝑥""𝑥"#𝑥"$𝑥#!𝑥#"𝑥##𝑥#$𝑥$!𝑥$"𝑥$#𝑥$$
𝑧##𝑧#!𝑧!#𝑧!!
𝑤!! 0 0 0𝑤!" 𝑤!! 0 0𝑤!# 𝑤!" 0 00 𝑤!# 0 0𝑤"! 0 𝑤!! 0𝑤"" 𝑤"! 𝑤!" 𝑤!!𝑤"# 𝑤"" 𝑤!# 𝑤!"0 𝑤"# 0 𝑤!#𝑤#! 0 𝑤"! 0𝑤#" 𝑤#! 𝑤"" 𝑤"!𝑤## 𝑤#" 𝑤"# 𝑤""0 𝑤## 0 𝑤"#0 0 𝑤#! 00 0 𝑤#" 𝑤#!0 0 𝑤## 𝑤#"0 0 0 𝑤##
𝑥!! 𝑥!" 𝑥!# 𝑥!$
𝑥"! 𝑥"" 𝑥"# 𝑥"$
𝑥#! 𝑥#" 𝑥## 𝑥#$
𝑥$! 𝑥$" 𝑥$# 𝑥$$
𝑤!! 𝑤!" 𝑤!#
𝑤"! 𝑤"" 𝑤"#
𝑤#! 𝑤#" 𝑤##
𝑧!! 𝑧!"
𝑧"! 𝑧""
=
=
𝑥!" = 𝑤!"𝑧## + 𝑤!!𝑧#! + 𝑤#"𝑧!# + 𝑤#!𝑧!!
∗%
Convolve input with flipped filter
Transpose Convolution
![Page 35: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/35.jpg)
Transpose Convolution
input
output
![Page 36: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/36.jpg)
D! × H/4 ×W/4 D! × H/4 ×W/4
D" × H/8 ×W/8
![Page 37: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/37.jpg)
Advanced Techniques in Segmentation
![Page 38: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/38.jpg)
U-Net
Ronneberger et al. 2015
![Page 39: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/39.jpg)
U-Net
https://phillipi.github.io/pix2pix/
![Page 40: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/40.jpg)
U-Net and FPN
Lin et al. 2017
![Page 41: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/41.jpg)
Hypercolumn and Skip Connection
Hariharan et al. 2015
![Page 42: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/42.jpg)
Hypercolumn and Skip Connection
Bansal et al. 2017
![Page 43: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/43.jpg)
Dilated convolutions
• Idea: instead of reducing spatial resolution of feature maps, use a large sparse filter
Dilation factor 1 Dilation factor 2 Dilation factor 3
![Page 44: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/44.jpg)
Dilated convolutions
• Increase the receptive field
input
output
![Page 45: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/45.jpg)
Visualizing Deep Networks using “Saliency map”
![Page 46: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/46.jpg)
CAM: Class Activation Maps
Zhou et al. 2016
![Page 47: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/47.jpg)
CAM: Class Activation Maps
![Page 48: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/48.jpg)
CAM: Class Activation Maps
![Page 49: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/49.jpg)
CAM: Class Activation Maps
![Page 50: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/50.jpg)
CAM: Class Activation Maps
https://www.youtube.com/watch?v=fZvOy0VXWAI&ab_channel=BoleiZhou
![Page 51: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/51.jpg)
Visualizing Deep Networks by maximizing activation
![Page 52: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/52.jpg)
Visualization by optimization
• We can synthesize images that maximize activation of a given neuron.
• Find image 𝑥 maximizing target activation 𝑓 𝑥 subject to natural image regularization penalty 𝑅(𝑥):
𝑥∗ = arg max" 𝑓 𝑥 − 𝜆𝑅(𝑥)
![Page 53: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/53.jpg)
Visualization by optimization• Maximize 𝑓 𝑥 − 𝜆𝑅(𝑥)
• 𝑓 𝑥 is score for a category before softmax• 𝑅(𝑥) is L2 regularization• Perform gradient ascent starting with zero image, add dataset mean
to result
Simonyan et al. 2014
![Page 54: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/54.jpg)
Visualization by optimization
Compute the Loss
BP the gradients, but do not train the network
Keep adding/aggregating the gradients under constraint 𝑅(𝑥)
![Page 55: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/55.jpg)
Visualization by optimization
![Page 56: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/56.jpg)
Google DeepDreamAmplify one layer instead of just one neuron.
Choose an image and a layer in a CNN; repeat:1. Forward: compute activations at chosen layer2. Set gradient of chosen layer equal to its activation
Equivalent to maximizing ∑6 𝑓6)(𝑥)3. Backward: Compute gradient w.r.t. image4. Update image (with some tricks)
![Page 57: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/57.jpg)
![Page 58: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/58.jpg)
![Page 59: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/59.jpg)
![Page 60: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/60.jpg)
![Page 61: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/61.jpg)
![Page 62: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/62.jpg)
![Page 63: Image Segmentation and Visualization · 2021. 6. 7. · Visualization by optimization •Maximize "!−-#(!) • !"is score for a category before softmax • #(")is L2 regularization](https://reader036.fdocuments.us/reader036/viewer/2022071605/6140eabc83382e045471c1a9/html5/thumbnails/63.jpg)
Next Class
Visualizing Deep Networks