Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For...
Transcript of Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For...
![Page 1: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/1.jpg)
Institute of Visual Computing
Introduction to Deep Learning
February 17, 2020
For slides credits we thank Vagia Tsiminaki.
Martin Oswald
![Page 2: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/2.jpg)
Institute of Visual Computing
Outline:
What is Deep Learning?
Artificial Neural Networks
Convolutional Neural Networks
Training Deep Learning Architectures
Applications on Computer Vision
![Page 3: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/3.jpg)
Institute of Visual Computing
What is Deep Learning?
Figure Credit: Adam Gibson, Josh Patterson “Deep Learning”
![Page 4: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/4.jpg)
Institute of Visual Computing
What is Deep Learning?
Machine Learning:
Input Data
Train model
Use trained model for new prediction
![Page 5: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/5.jpg)
Institute of Visual Computing
Towards Deep Learning
Hand-crafted Features
E.g. Canny edges, Harris corners, SIFT
Feature Learning
Extract automatically patterns (features)
Deep Learning
Learning hierarchical representations of data
End to end learning
![Page 6: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/6.jpg)
Institute of Visual Computing
Image Classification
Chihuahua or Muffin ?
f( ) = ”Muffin” f( ) = ”Chihuahua”
![Page 7: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/7.jpg)
Institute of Visual Computing
Image Classification
Machine Learning:
Input Data: Training set of images
and labels
Train model (Image classifier)
Use trained model for new prediction
![Page 8: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/8.jpg)
Institute of Visual Computing
Image Classification
Training Images
TrainingImage
features
Image
classifier
Image
labels
Slide Credit:CS 131, Lecture 1, 2016
Training
Testing
Image
features
Learned image
classifier”Chihuahua”
![Page 9: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/9.jpg)
Institute of Visual Computing
Image Classification
Training Images
TrainingImage
features
Image
classifier
Image
labels
Slide Credit:CS 131, Lecture 1, 2016
Training
Testing
Image
features
Learned image
classifier”Chihuahua”
Feature Engineering
![Page 10: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/10.jpg)
Institute of Visual Computing
Image Classification
Training Images
Image
labels
Slide Credit:CS 131, Lecture 1, 2016
Testing
”Chihuahua”
Training
Image
features
Image
classifier
Learned Model
Learned
features
Learned
classifierLearned Model
Learned
features
Learned
classifier
Feature Learning
![Page 11: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/11.jpg)
Institute of Visual Computing
Image Classification
Training
Images
Image
labels
Learned ModelDeep Learning
Training
Low-level
features
Image
Classifier
High-level
features
Mid-level
features
Mid-level
features
Low-level
features
Image
Classifier
High-level
features
![Page 12: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/12.jpg)
Institute of Visual Computing
Artificial Neural Networks
Figure Credit: Artificial Intelligence Techniques for Modelling of Temperature in the Metal
Cutting Process
Input Layer
Hidden Layer
Output Layer
![Page 13: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/13.jpg)
Institute of Visual Computing
Artificial Neuron
x1, x2,…, xN: Inputs to the neuron
w0,w1, w2,…,wN: Weights on each
input
f: Activation function
![Page 14: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/14.jpg)
Institute of Visual Computing
Activation Function
Sigmoid Function Tanh Activation
ReLU (Rectified Linear Unit)
![Page 15: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/15.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Convolution Layer
Rectified Linear Unit (ReLu)
Pooling Layer
Fully Connected Layer
![Page 16: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/16.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Image Source:http://deeplearning.stanford.edu/wiki/index.php/Feature_extraction_using_convolution
Image
Filter or
KernelConvolution
Feature Map or
Activation Map or
Convolved Feature
Convolution layer to extract features from input
image
![Page 17: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/17.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Image Source: https://ujjwalkarn.me/2016/08/11/intuitive-explanationconvnets/
Convolution layer to extract features from input
image
![Page 18: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/18.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Image Source: https://cs.nyu.edu/~fergus/tutorials/deep_learning_cvpr12/
Convolution layer to extract features from input
image
![Page 19: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/19.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Image Feature Map
Size of Feature Map
Depth: Number of filters
Stride
Zero-padding
Convolution layer to extract features from input
image
![Page 20: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/20.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Rectified Linear Unit (ReLu) to introduce non-
linearity
Most of real-world data are non linear
Convolution is linear operation
Image Source:http://mlss.tuebingen.mpg.de/2015/slides/fergus/Fergus_1.pdf
Output = max (0, Input)
![Page 21: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/21.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Pooling Layer
Max
Average
Sum
Image Source:http://mlss.tuebingen.mpg.de/2015/slides/fergus/Fergus_1.pdf
![Page 22: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/22.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Image Source:http://cs231n.github.io/convolutional-networks/
Pooling Layer to:
Reduce the dimension of input
Reduce the number of parameters and
computations(control overfitting)
Make the network invariant to small
transformations, distortions, translations
Get and almost scale invariant representation of
input
![Page 23: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/23.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Fully Connected Layer
Each node is connected to each node in the
adjacent layer
Input: High-level features of the input image
from the convolutional and pooling layers
Goal:
Classification
Regression
Segmentation
![Page 24: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/24.jpg)
Institute of Visual Computing
Convolutional Neural Networks
Fully Connected Layer
Each node is connected to each node in the
adjacent layer
Input: High-level features of the input image
from the convolutional and pooling layers
Goal:
Classification
Multi Layer Perceptron with a softmax activation function in
the output layer
![Page 25: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/25.jpg)
Institute of Visual Computing
Image Classification
Chihuahua or Muffin ?
f( ) = ”Muffin” f( ) = ”Chihuahua”
![Page 26: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/26.jpg)
Institute of Visual Computing
Image Classification
Training
Images
Image
labels
Learned ModelDeep Learning
Training
Low-level
features
Image
Classifier
High-level
features
Mid-level
features
Mid-level
features
Low-level
features
Image
Classifier
High-level
features
![Page 27: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/27.jpg)
Institute of Visual Computing
Training Deep Learning Architectures
Classes= {”Chihuahua”, ” Muffin”}
Input= ”Chihuahua”
Target Vector= [1,0]
Chihuahua(0)
Muffin(1)
![Page 28: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/28.jpg)
Institute of Visual Computing
Training Deep Learning Architectures
Initialize weights of filters
Forward propagation pass
Convolution
ReLu
Pooling
Fully connected layer
Chihuahua(0)
Muffin(1)
![Page 29: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/29.jpg)
Institute of Visual Computing
Training Deep Learning Architectures
Chihuahua(0)
Muffin(1)
Initialize weights of filters
Forward propagation pass
Calculate the total error: 𝐸 = Σ1
2(𝑡𝑎𝑟𝑔𝑒𝑡 − 𝑜𝑢𝑡𝑝𝑢𝑡)2
Backward propagation pass
![Page 30: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/30.jpg)
Institute of Visual Computing
Training Deep Learning Architectures
Chihuahua(0)
Muffin(1)
Calculate the total error: 𝐸 = Σ1
2(𝑡𝑎𝑟𝑔𝑒𝑡 − 𝑜𝑢𝑡𝑝𝑢𝑡)2
Backward propagation pass
Iterate Forward – Backward propagation with all training data
![Page 31: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/31.jpg)
Institute of Visual Computing
Training Deep Learning Architectures
Chihuahua(0)
Muffin(1)
Calculate the total error: 𝐸 = Σ1
2(𝑡𝑎𝑟𝑔𝑒𝑡 − 𝑜𝑢𝑡𝑝𝑢𝑡)2
Backward propagation pass
Iterate Forward – Backward propagation with all training data
![Page 32: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/32.jpg)
Institute of Visual Computing
Applications on Computer Vision
Super-Resolution
Figure Credit: Ledig et al. “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”
Image Segmentation &
Classification
Figure Credit: Dai, He and Sun “Instance-aware Semantic Segmentation via Multi-task Network Cascades”
Style Transfer
Figure Credit: Zhu et al. “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks ”
Semantic 3D Reconstruction
Figure Credit: Dai et al. “3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation”, ECCV 2018
![Page 33: Introduction to Deep Learning - CVG @ ETHZ · Introduction to Deep Learning February 17, 2020 For slides credits we thank Vagia Tsiminaki. Martin Oswald. Institute of Visual Computing](https://reader033.fdocuments.us/reader033/viewer/2022050409/5f8693f9fe512d1f74782d9a/html5/thumbnails/33.jpg)
Institute of Visual Computing
Questions ?