Neural Network Background - GitHub Pages
Transcript of Neural Network Background - GitHub Pages
![Page 1: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/1.jpg)
Neural Network
BackgroundZhifei Zhang and Hairong Qi
![Page 2: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/2.jpg)
3/29/18 2
Algorithms whose performance improve as they are exposed to more data over time
A program that can sense, reason, act, and adapt
Multi-layer neural networks learn from vast amounts of data
• Naive Bayes• kNN• Decision Tree• SVM• K-Means• Dimensionality Reduction,
e.g., FLD, PCA
![Page 3: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/3.jpg)
3/29/18 3For more detailed history of deep learning https://www.import.io/post/history-of-deep-learning/
![Page 4: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/4.jpg)
3/29/18 4
Intuition of Neural Network
https://www.quora.com/What-is-an-intuitive-explanation-for-neural-networks
![Page 5: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/5.jpg)
3/29/18 5
Perceptron
• Weights shows the strength of the particular node.
• A bias value allows you to shift the activation function to the left or right.
• The activation function map the input between the required values, e.g., [0, 1].
Perceptron is usually used to classify the data into two parts. Therefore, it is also known as a Linear Binary Classifier.
Activation Function
WeightedSum
Bias
ActivationFunction
![Page 6: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/6.jpg)
3/29/18 6
Perceptron
WeightedSum
Bias
ActivationFunction
0
1 1
1 1
1 T
n n
n n
wx w
sx wx w- -
æ ö æ öç ÷ ç ÷ç ÷ ç ÷ç ÷ ç ÷´ =ç ÷ ç ÷ç ÷ ç ÷ç ÷ ç ÷è ø è ø
! !
Common activation functions:
ˆ( )F s y=
![Page 7: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/7.jpg)
3/29/18 7
Perceptron Learning Rule
WeightedSum
Bias
ActivationFunction
0
1 1
1 1
1 T
n n
n n
wx w
sx wx w- -
æ ö æ öç ÷ ç ÷ç ÷ ç ÷ç ÷ ç ÷´ =ç ÷ ç ÷ç ÷ ç ÷ç ÷ ç ÷è ø è ø
! !
!" ≔ !" − & '( − ' )"
* = 12 ' − '( .
∆!" = 01023
= 0104(
04(05
05023
Objective:
Partial derivative of * w.r.t. !":
∆!" = '( − ' 6 7′(:) 6 )"
Where < is the learning rate that is a positive value in the range of (0, 1).
Assume the input = = (=>, … , =A), ground truth B, and output of perceptron BC, the weight is updated as follow:
Assume 1, directly pass the error backward
ˆ( )F s y=
![Page 8: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/8.jpg)
3/29/18 8
XOR Problem
![Page 9: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/9.jpg)
3/29/18 9
Multi-layer Perceptron --- Neural Network
Hidden layerInput layer Output layer
Neural Network in more practice
![Page 10: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/10.jpg)
3/29/18 10
Online demo: https://lecture-demo.ira.uka.de/neural-network-demo/
Multi-layer Perceptron --- Solve the XOR Problem
![Page 11: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/11.jpg)
3/29/18 11
BackpropagationHow the multi-layer perceptron (neural network) is learned?
Back propagate the error layer-by-layer ℎ"
#$
%" & = 12 # − #$ +
Output error:,-
%-
![Page 12: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/12.jpg)
3/29/18 12
BackpropagationHow those multi-layer perceptron (neural network) is learned?
Back propagate the error layer-by-layer
∆"# = %& − % ( )′(,-) ( ℎ#
ℎ#
%&
"# 0 = 12 % − %& 3
Output error:45
"5
![Page 13: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/13.jpg)
3/29/18 13
BackpropagationHow those multi-layer perceptron (neural network) is learned?
Back propagate the error layer-by-layer, and
!"
# = 12 ! − !" (
Output error:)*
∆,* = - . /′(23) . )*-
Hidden nodes are independent from each other
- = 5#5ℎ7
= 5#5!"
5!"528
525ℎ7
,*
- = !" − ! . /′(28) . ,7
ℎ7
,7
∆,7 = !" − ! . /′(28) . ℎ7
Back propagated error:
![Page 14: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/14.jpg)
3/29/18 14
Convolutional Neural Network (CNN)
Handle images which need to preserve the special structure
![Page 15: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/15.jpg)
3/29/18 15
Convolutional Neural Network (CNN)
LeNet-5 by Yann LeCun et al., 1998
A demo code in Matlab: https://github.com/ZZUTK/An-Example-of-CNN-on-MNIST-dataset
![Page 16: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/16.jpg)
3/29/18 16
Appearance of CNN is early (back to 1998) but it booms around 2010s, WHY?
Convolutional Neural Network (CNN)
From 2009 From 2010
Large dataset (over 10 million) Powerful computing unit
![Page 17: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/17.jpg)
3/29/18 17
Convolutional Neural Network (CNN)
Appearance of CNN is early (back to 1998) but it booms around 2010s, WHY?
Deep convolutional neural networks with competitive performance to human
AlexNet, 2012
![Page 18: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/18.jpg)
3/29/18 18
Convolutional Neural Network (CNN)
Since 2010s, the deep convolutional neural networks are mostly referred to as deep learning, and it flourishes in computer vision area until today.
![Page 19: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/19.jpg)
3/29/18 19
Convolutional Neural Network (CNN)Convolution and Pooling
Receptive field
Where is the activation function?
![Page 20: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/20.jpg)
3/29/18 20
Convolutional Neural Network (CNN)
Why deep? What does each layer learn?
![Page 21: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/21.jpg)
3/29/18 21
Convolutional Neural Network (CNN)
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) results show deep learning surpassing human levels of accuracy
![Page 22: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/22.jpg)
3/29/18 22
Convolutional Neural Network (CNN)
How to learn a CNN? Again, Backpropagation.
The same as multi-layer perceptronNo parameters, do not need update. Directly pass the error backward.
![Page 23: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/23.jpg)
3/29/18 23
!
Convolutional Neural Network (CNN)
How to learn a CNN? Again, Backpropagation.Multi-layer Perceptron Convolutional Neural Network
" =
∆! = % & ' ∆" = '()*+,- ∗ %
For simplicity, assume /′ 1 = 2 here.
' '
% %
https://github.com/ZZUTK/An-Example-of-CNN-on-MNIST-dataset/blob/master/doc/Derivation_of_Backpropagation_in_CNN.pdf
3 = % & ! 3 = % ∗ "()*+,-
![Page 24: Neural Network Background - GitHub Pages](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271d54c5c327a08724a7565/html5/thumbnails/24.jpg)
3/29/18 24