Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei...
Transcript of Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei...
![Page 1: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/1.jpg)
Edge Detection: Deep Contour
Wei Shen’s work – uses clustering and patches.
![Page 2: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/2.jpg)
Our Work2
DeepContour: A Deep Convolutional Feature Learned by Positive-sharing
Loss for Contour Detection
Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, Zhijiang Zhang
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015
![Page 3: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/3.jpg)
Motivation3
Why to apply CNN for contour detection? Contour is hard to define
Contour data are sufficient for CNN training (millions of local contour patches)
![Page 4: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/4.jpg)
Problem Formulation4
Given a color image patch 𝑥𝑥 ∈ ℝ𝑛𝑛×𝑛𝑛×3, our goal is to determine whether its center pixel is passed through by contours or not.
𝑥𝑥 ∈ ℝ𝑛𝑛×𝑛𝑛×3 → 𝑧𝑧 ∈ 0,1
Q: Is a good idea to directly use CNN as a blackbox to address this binary classification problem?
Non-contour Contour
![Page 5: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/5.jpg)
Obstacle5
The large variations in the contour shapes
Solution: Partitioning contour patches into compact clusters to convert the binary classification problem to a multi-class classification problem
Ref: J. J. Lim, C. L. Zitnick, and P. Dollár. Sketch tokens: A learned mid-level representation for contour and object detection. CVPR, 2013.
![Page 6: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/6.jpg)
Obstacle6
How to define the loss function?Q: Is softmax a good choice?
Softmax function penalizes the loss of each class equally
The losses for contour versus non-contour should be emphasized
Solution: Adding a regularized term to focus on the end goal of binary classification
![Page 7: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/7.jpg)
7
Data Preparation
Pre-cluster contour patches according to their contour shapes.
Assign a label 𝑦𝑦 to each contour patch 𝑥𝑥 according to the pre-cluster index 1, …𝐾𝐾 .
𝑥𝑥 ∈ ℝ𝑛𝑛×𝑛𝑛×3 → 𝑦𝑦 ∈ 0,1, …𝐾𝐾
Negative Positive
![Page 8: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/8.jpg)
8
Method
CNN Architecture
Loss FunctionLet (𝑎𝑎𝑗𝑗
𝑖𝑖 ; 𝑗𝑗 = 1, …𝐾𝐾) be the output of unit 𝑗𝑗 in FC2 for a image patch 𝑥𝑥𝑗𝑗
(𝑖𝑖), the probability that the label is 𝑗𝑗 is
𝑝𝑝𝑗𝑗(𝑖𝑖) =
exp(𝑎𝑎𝑙𝑙(𝑖𝑖))
∑𝑙𝑙=0𝐾𝐾 exp(𝑎𝑎𝑙𝑙(𝑖𝑖))
![Page 9: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/9.jpg)
9
Method
𝐽𝐽
= −1𝑚𝑚�𝑖𝑖=1
𝑚𝑚
�𝑗𝑗=0
𝐾𝐾
𝟏𝟏 𝑦𝑦 𝑖𝑖 = 𝑗𝑗 log𝑝𝑝𝑗𝑗𝑖𝑖
−1𝑚𝑚 �
𝑖𝑖=1
𝑚𝑚
𝜆𝜆 𝟏𝟏 𝑦𝑦 𝑖𝑖 = 0 log𝑝𝑝0𝑖𝑖 + �
𝑗𝑗=1
𝐾𝐾
𝟏𝟏 𝑦𝑦 𝑖𝑖 = 𝑗𝑗 log(1 − 𝑝𝑝0𝑖𝑖 )
Softmax loss
Positive-sharing loss, the loss for positive class is shared among each pre-clustered contour classes
![Page 10: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/10.jpg)
10
Method
To apply standard back-propagation to optimize the parameters of the network
𝜕𝜕𝐽𝐽
𝜕𝜕𝑎𝑎0(𝑖𝑖) =
1𝑚𝑚
𝜆𝜆 + 1 𝟏𝟏 𝑦𝑦 𝑖𝑖 = 0 𝑝𝑝0𝑖𝑖 − 1 + (𝜆𝜆 + 1)�
𝑗𝑗=1
𝐾𝐾
𝟏𝟏 𝑦𝑦 𝑖𝑖 = 𝑗𝑗 𝑝𝑝0𝑖𝑖
𝜕𝜕𝐽𝐽
𝜕𝜕𝑎𝑎𝑙𝑙(𝑖𝑖) =
1𝑚𝑚
𝜆𝜆𝟏𝟏 𝑦𝑦 𝑖𝑖 = 0 + 1 𝑝𝑝𝑙𝑙(𝑖𝑖) − 𝟏𝟏 𝑦𝑦 𝑖𝑖 = 𝑙𝑙 − 𝜆𝜆�
𝑗𝑗=1
𝐾𝐾
𝟏𝟏 𝑦𝑦 𝑖𝑖 = 𝑗𝑗𝑝𝑝0𝑖𝑖 𝑝𝑝𝑙𝑙
𝑖𝑖
1 − 𝑝𝑝0𝑖𝑖
![Page 11: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/11.jpg)
11
Method
CNN model validation
𝛾𝛾 ∈ [−1,1], measuring the discrimination of the learned model between positive and negative samples.
The learned features of FC1 will be fed into structured forest to perform contour detection.
𝛾𝛾 =1𝑚𝑚�𝑖𝑖=1
𝑚𝑚
𝟏𝟏 𝑦𝑦 𝑖𝑖 = 0 − 𝟏𝟏 𝑦𝑦 𝑖𝑖 > 0 (𝑝𝑝0𝑖𝑖 − (1 − 𝑝𝑝0
𝑖𝑖 ))
![Page 12: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/12.jpg)
12
Experimental results
Deep Feature Visualization
Positive-sharing loss
Local gradient
![Page 13: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/13.jpg)
Experimental results
13
Results on BSDS500
![Page 14: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/14.jpg)
14
Experimental results
Results on BSDS500
![Page 15: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/15.jpg)
15
Experimental results
Results on NYUD
![Page 16: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/16.jpg)
16
Experimental results
Cross Dataset Generalization
DeepContour ODS OIS AP
BSDS/BSDS .76 .78 .80
NYU/BSDS .72 .74 .77
BSDS/NYU .59 .60 .53
NYU/NYU .62 .63 .57
SE [11] ODS OIS AP
BSDS/BSDS .74 .76 .78
NYU/BSDS .72 .73 .76
BSDS/NYU .55 .57 .46
NYU/NYU .60 .61 .56
SCG [39] ODS OIS AP
NYU/NYU .55 .57 .46
gPb [2] ODS OIS AP
NYU/NYU .51 .52 .37
![Page 17: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/17.jpg)
17
Experimental results
Parameter Discussion
![Page 18: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/18.jpg)
Outline18
Contour Detection Overview Milestones
Our Work
Discussions
![Page 19: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/19.jpg)
Discussions19
Speed Caffe – per-patch mean subtraction is the bottleneck
Accuracy Limitation may caused by the confusing labels
![Page 20: Edge Detection: Deep Contourayuille/.../DeepNetsEdgeDetection.pdfEdge Detection: Deep Contour Wei Shen’s work – uses clustering and patches. Our Work 2 DeepContour: A Deep Convolutional](https://reader034.fdocuments.us/reader034/viewer/2022052100/60395817d4de4267ad423edd/html5/thumbnails/20.jpg)
Discussions20
Thank you! Q&A