CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural...
Transcript of CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural...
![Page 1: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/1.jpg)
CSCE636NeuralNetworks(DeepLearning)
Lecture2:Mathematical BuildingBlocksofNeuralNetworks
Anxiao (Andrew)Jiang
![Page 2: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/2.jpg)
Chapter2
Beforewebegin:themathematicalbuildingblocksofneuralnetworks
![Page 3: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/3.jpg)
Whataneuralnetworkdoes:learnafunction
NeuralNetworkx
valueoff(x)
Theneuralnetworklearnsthefunctionf(x),eitherexactlyorapproximately.
![Page 4: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/4.jpg)
Application:HandwrittenDigitRecognition
NeuralNetwork 4
Task:Classifygrayscaleimagesofhandwrittendigits(28x28pixels)intotheir10categories(0through9).
![Page 5: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/5.jpg)
Howtostart?
![Page 6: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/6.jpg)
Step1:Loadthedataset
MNISTDataset:60,000 trainingimagesand10,000 testimages,alongwiththeirlabels.
![Page 7: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/7.jpg)
Step1:Loadthedataset
train_images: 60,000x28x28array,whereeachelement (pixel) isaninteger in[0,255]train_labels: vectoroflength60,000,whereeachelement (label) isaninteger in[0,9]
test_images: 10,000x28x28array,whereeachelement (pixel) isaninteger in[0,255]test_labels: vectoroflength10,000,whereeachelement (label)isaninteger in[0,9]
Rule:trainingdataandtestdataaredisjoint.Onlyusetrainingdatatotrainneuralnetwork!
![Page 8: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/8.jpg)
4
![Page 9: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/9.jpg)
Step2:Buildneuralnetworkarchitecture
428x282-d array
0
1
2
28x28-1 =783
0
1
2
511
512neurons
![Page 10: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/10.jpg)
Whatisaneuron
Activation function(manypossible forms)
ReLU (mostpopularActivationfunction)
![Page 11: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/11.jpg)
Step2:Buildneuralnetworkarchitecture
428x282-d array
0
1
2
28x28-1 =783
0
1
2
511
512neurons
0
1
2
9
10neurons
![Page 12: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/12.jpg)
Softmax (popularactivationfunctionforthelastlayerofaclassificationnetwork)
![Page 13: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/13.jpg)
Step2:Buildneuralnetworkarchitecture
428x282-d array
0
1
2
28x28-1 =783
0
1
2
511
512neurons
0
1
2
9
10neurons
Probability oflabel “0”
Probability oflabel “1”
Probability oflabel “2”
Probability oflabel “9”
![Page 14: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/14.jpg)
Step3:chooselossfunction,optimizer,andtargetmetrics
![Page 15: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/15.jpg)
Categoricalcross-entropy(apopularlossfunctionformulti-classclassification)
Numberofsamples
Numberofclasses
Trueprobability(1or0)thisinputbelongstoclassj
probabilitypredictedbyneuralnetworkthatthisinputbelongsto
classj
![Page 16: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/16.jpg)
RMSProp (apopularoptimizer,detailstobeintroduced later)
![Page 17: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/17.jpg)
Accuracy: fractionoftimesthattheneuralnetworkmakescorrectionpredictions• Ifwecareaboutaccuracy,whydoweoptimizecategoricalcross-entropyduringtraining?• Answer:lossfunctionneedstobedifferentiable.(Andthelossfunctioniscloselyrelatedtothetargetmetric.Minimizingthelossfunctionis(approximatelyorprecisely)optimizingthetargetmetric.
![Page 18: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/18.jpg)
428x282-d array
0
1
2
28x28-1 =783
0
1
2
511
512neurons
0
1
2
9
10neurons
Probability oflabel “0”
Probability oflabel “1”
Probability oflabel “2”
Probability oflabel “9”
The“Teacher”:
Loss function: categoricalcross-entropyOptimizer: RMSPropTargetMetric:Accuracy
![Page 19: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/19.jpg)
Step4:PreparetrainingandtestdataHere:Reshapeandnormalizeinputtrainingdata
train_images:Originally: 3-dimensional arrayofsize60000 x28x28,whereeach element isanintegerin [0,255]Afterreshaping: 2-dimensional arrayofsize60000 x784,whereeach element isanintegerin [0,255]Afternormalization: 2-dimensional arrayofsize60000 x784,where eachelement is arealnumber in[0,1]
28x282-d array
1-d array oflength28*28 =784
Normalize values inthearraytobetween0and 1
![Page 20: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/20.jpg)
428x282-d array
0
1
2
0
1
2
511
512neurons
0
1
2
9
10neurons
Probability oflabel “0”
Probability oflabel “1”
Probability oflabel “2”
Probability oflabel “9”
The“Teacher”:
Loss function: categoricalcross-entropyOptimizer: RMSPropTargetMetric:Accuracy
783
0
1
2
783
![Page 21: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/21.jpg)
“Reshape”outputtrainingdata:categoricallyencodeeachlabelusingone-hotencoding
Label One-hotencoding0 1,0,0,0,0,0,0,0,0,01 0,1,0,0,0,0,0,0,0,02 0,0,1,0,0,0,0,0,0,03 0,0,0,1,0,0,0,0,0,04 0,0,0,0,1,0,0,0,0,0
Label One-hotencoding5 0,0,0,0,0,1,0,0,0,06 0,0,0,0,0,0,1,0,0,07 0,0,0,0,0,0,0,1,0,08 0,0,0,0,0,0,0,0,1,09 0,0,0,0,0,0,0,0,0,1
![Page 22: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/22.jpg)
428x282-d array
0
1
2
0
1
2
511
512neurons
0
1
2
9
10neurons
The“Teacher”:
Loss function: categoricalcross-entropyOptimizer: RMSPropTargetMetric:Accuracy
783
0
1
2
783
0
1
2
9
![Page 23: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/23.jpg)
Step5:Traintheneuralnetwork
428x282-d array
0
1
2
0
1
2
511
512neurons
0
1
2
9
10neurons
The“Teacher”:Loss function, Optimizer, TargetMetric:Accuracy
783
0
1
2
783
0
1
2
9
![Page 24: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/24.jpg)
Batchsize:thenumberofsamplestouseeachtimeforcomputingthelossfunctionandupdatingtheweights.
Epochs:thenumberoftimesthetrainingprocessusesthewholetrainingdataset.
![Page 25: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/25.jpg)
Andsoon(totally5epochs).
Accuracyontrainingdata:97.8%
![Page 26: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/26.jpg)
Step6:Testthetrainedneuralnetwork
Comparetotrainingaccuracy:0.989
Test accuracyis(clearly) lowerthantrainingaccuracy.Maybethere issome over-fittingtodata.
Butstill, performance isnice!
![Page 27: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/27.jpg)
Summary
![Page 28: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/28.jpg)
![Page 29: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/29.jpg)
Step1:Loadthedataset
4
![Page 30: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/30.jpg)
Step2:Buildneuralnetworkarchitecture
428x282-d array
0
1
2
28x28-1 =783
0
1
2
511
0
1
2
9
![Page 31: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/31.jpg)
Step3:chooselossfunction,optimizer,andtargetmetrics
428x282-d array
0
1
2
28x28-1 =783
0
1
2
511
0
1
2
9
Probability oflabel “0”
Probability oflabel “1”
Probability oflabel “2”
Probability oflabel “9”
The“Teacher”:Loss function, Optimizer, TargetMetric
![Page 32: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/32.jpg)
Step4:Preparetrainingandtestdata
428x282-d array
0
1
2
0
1
2
511
0
1
2
9783
0
1
2
783
0
1
2
9
The“Teacher”:Loss function, Optimizer, TargetMetric
![Page 33: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/33.jpg)
Step5:Traintheneuralnetwork
428x282-d array
0
1
2
0
1
2
511
512neurons
0
1
2
9
10neurons
The“Teacher”:Loss function, Optimizer, TargetMetric:Accuracy
783
0
1
2
783
0
1
2
9
![Page 34: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/34.jpg)
Step6:Testthetrainedneuralnetwork
HowdidIdo?
Well…
![Page 35: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/35.jpg)
MiscellaneousBasicConcepts
![Page 36: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/36.jpg)
Datarepresentation:Tensor(Array)
• Scalarnumbers(0-dimensionaltensors)• Vectors(1-dtensors)• Matrices(2-dtensors)• 3-dtensors,andhigher-dimensionaltensors
• Keyattributesforatensor:• (1)numberofaxes• (2)shape• (3)datatype
![Page 37: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/37.jpg)
Somebasictensoroperations
• Addtwotesnors (ofthesameshape):element-wiseaddition• ApplyaReLU activationfunctiontoatensor:element-wiseoperation
• TensorProduct(alsocalledtensordot)
![Page 38: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/38.jpg)
Reshapetensor
![Page 39: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/39.jpg)
Basictermsforaneuralnetwork
• Layers:thebuildingblocksinaneuralnetwork• Model:networkoflayers• Lossfunctionandoptimizer:keystoconfiguringthelearningprocess
![Page 40: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/40.jpg)
Keras:adeeplearninglibraryforPython
PyTorchisgettingpopulartoday
![Page 41: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/41.jpg)
Keras:adeeplearninglibraryforPython
UseaGPUwhenpossible
![Page 42: CSCE 636 Neural Networks (Deep Learning)What a neural network does: learn a function x Neural Network value of f(x) The neural network learns the function f(x), either exactly or approximately.](https://reader033.fdocuments.us/reader033/viewer/2022053023/6054c92d703b9049da213ea1/html5/thumbnails/42.jpg)
Jupyter notebook:anicewaytoeditandrundeeplearningexperiments