One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z...
Transcript of One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z...
![Page 1: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/1.jpg)
deeplearning.ai
One hidden layerNeural Network
Neural NetworksOverview
![Page 2: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/2.jpg)
Andrew Ng
What is a Neural Network?
!"!#!$
%&
x
w
b
' = )*! + , - = .(') ℒ(-, %)
x
4["]
,["]'["] = 4["]! + ,["] -["] = .('["]) '[#] = 4[#]-["] + ,[#] -[#] = .('[#]) ℒ(-[#], %)
!"!#!$
%&
4[#]
,[#]
![Page 3: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/3.jpg)
deeplearning.ai
One hidden layerNeural Network
Neural NetworkRepresentation
![Page 4: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/4.jpg)
Andrew Ng
Neural Network Representation
!"
!#
!$
%&
![Page 5: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/5.jpg)
deeplearning.ai
One hidden layerNeural Network
Computing aNeural Network’s
Output
![Page 6: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/6.jpg)
Andrew Ng
Neural Network Representation
!"!#!$
%&
' = )!! + +
)!! + +,
!"!#!$
-(') , = %&'
, = -(')
![Page 7: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/7.jpg)
Andrew Ng
! = #!$ + &
#!$ + &'
$($)$*
+(!) ' = ./!
' = +(!)
Neural Network Representation
$($)$*
./
$($)$*
./
![Page 8: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/8.jpg)
Andrew Ng
Neural Network Representation
!"!#!$
%&
'""
'#"
'$"
'("
)"" = +"" ,! + /"["], '"["] = 3()"" )
)#" = +#" ,! + /#
["], '#["] = 3()#" )
)$" = +$" ,! + /$["], '$["] = 3()$" )
)(" = +(" ,! + /(
["], '(["] = 3()(" )
![Page 9: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/9.jpg)
Andrew Ng
Neural Network Representation learningGiven input x:
! " = $ " % + ' "
( " = )(! " )
! , = $ , ( " + ' ,
( , = )(! , )
%"%,%-
./
(""
(,"
(-"
(0"
![Page 10: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/10.jpg)
deeplearning.ai
One hidden layerNeural Network
Vectorizing across multiple examples
![Page 11: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/11.jpg)
Andrew Ng
Vectorizing across multiple examples
!"!#!$
%&
' " = ) " ! + + "
, " = -(' " )' # = ) # , " + + #
, # = -(' # )
![Page 12: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/12.jpg)
Andrew Ng
! " ($) = ' " (($) + * "
+ " ($) = ,(! " $ )! - ($) = ' - + " ($) + * -
+ - ($) = ,(! - $ )
Vectorizing across multiple examplesfor i = 1 to m:
![Page 13: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/13.jpg)
deeplearning.ai
One hidden layerNeural Network
Explanation for vectorized
implementation
![Page 14: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/14.jpg)
Andrew Ng
Justification for vectorized implementation
![Page 15: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/15.jpg)
Andrew Ng
!"!#!$
%&
Recap of vectorizing across multiple examplesfor i = 1 to m
' " ()) = , " !()) + . "
/ " ()) = 0(' " ) )' # ()) = , # / " ()) + . #
/ # ()) = 0(' # ) )…1 = !(") !(#) !(2)
/["](#)A["] = /["](") /["](2)…
6 " = , " 1 + . "
7 " = 0(6 " )6 # = , # 7 " + . #
7 # = 0(6 # )
![Page 16: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/16.jpg)
deeplearning.ai
One hidden layerNeural Network
Activation functions
![Page 17: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/17.jpg)
Andrew Ng
Activation functions
!"!#!$
%&
' " = ) " ! + + "
, " = -(' " )' # = ) # , " + + #
, # = -(' # )
Given x:
![Page 18: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/18.jpg)
Andrew Ng
Pros and cons of activation functionsa
zsigmoid: ! = 1
1 + &'(
z
a
x
a
z
a
![Page 19: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/19.jpg)
deeplearning.ai
One hidden layerNeural Network
Why do you need non-linear
activation functions?
![Page 20: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/20.jpg)
Andrew Ng
Activation function
! " = $ " % + ' "
( " = )["](! " )! . = $ . ( " + ' .
( . = )[.](! . )
Given x:
%"%.%/
01
![Page 21: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/21.jpg)
deeplearning.ai
One hidden layerNeural Network
Derivatives of activation functions
![Page 22: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/22.jpg)
Andrew Ng
Sigmoid activation function
a
z
!(#) = 11 + )*+
![Page 23: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/23.jpg)
Andrew Ng
!(#) = tanh(#)
Tanh activation functiona
z
![Page 24: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/24.jpg)
Andrew Ng
zReLU
a
zLeaky ReLU
a
ReLU and Leaky ReLU
![Page 25: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/25.jpg)
deeplearning.ai
One hidden layerNeural Network
Gradient descent forneural networks
![Page 26: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/26.jpg)
Andrew Ng
Gradient descent for neural networks
![Page 27: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/27.jpg)
Andrew Ng
Formulas for computing derivatives
![Page 28: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/28.jpg)
deeplearning.ai
One hidden layerNeural Network
Backpropagationintuition (Optional)
![Page 29: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/29.jpg)
Andrew Ng
Computing gradients
Logistic regression
! = #$% + '%#'
) = *(!) ℒ(), /)
![Page 30: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/30.jpg)
Andrew Ng
![#] = &[#]' + )[#]'
&["]
)["]
+[#] = ,(![#]) ℒ(+[0], y)![0] = &[0]' + )[0] +[0] = ,(![0])
Neural network gradients&[$]
)[$]
![Page 31: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/31.jpg)
Andrew Ng
!"[$] = !'[$]( ) *
!+[$] = !'[$]
!'[)] = " $ ,!'[$] ∗ .[)]′(z ) )
!"[)] = !'[)]3,
!+[)] = !'[)]
Summary of gradient descent!'[$] = ([$] − 5
![Page 32: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/32.jpg)
Andrew Ng
!"[$] = '[$] − )
!*[$] = !"[$]' + ,
!-[$] = !"[$]
!"[+] = * $ .!"[$] ∗ 0[+]′(z + )
!*[+] = !"[+]5.
!-[+] = !"[+]
!6["] = 7["] − 8
!*["] = 1:!6["]7 $ ,
!-["] = 1:;<. >?:(!6 " , '5A> = 1, BCC<!A:> = DE?C)
!6[$] = * " %!6["] ∗ 0[$]′(Z $ )
!*[$] = 1:!6[$]G%
!-[$] = 1:;<. >?:(!6 $ , '5A> = 1, BCC<!A:> = DE?C)
Summary of gradient descent
![Page 33: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/33.jpg)
deeplearning.ai
One hidden layerNeural Network
Random Initialization
![Page 34: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/34.jpg)
Andrew Ng
What happens if you initialize weights to zero?
!![!]"#
!$[!]"$%&!!
[$]
![Page 35: One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z Leaky ReLU a ReLU and Leaky ReLU deeplearning.ai One hidden layer Neural Network](https://reader035.fdocuments.us/reader035/viewer/2022071219/6054cd7085f80c117b52f22e/html5/thumbnails/35.jpg)
Andrew Ng
Random initialization
!![!]"#
!$[!]"$%&!!
[$]