Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12....
46
CS839: Probabilistic Graphical Models Lecture 16: Spectral Algorithms for GMs Theo Rekatsinas 1
Transcript of Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12....
CS839:ProbabilisticGraphicalModels
Lecture16:SpectralAlgorithmsforGMsTheoRekatsinas
1
Overview
2
• AnoverviewoftheDLcomponents• Historicalremarks:earlydaysofneuralnetworks• Modernbuildingblocks:units,layers,activationsfunctions,lossfunctions,etc.• Reverse-modeautomaticdifferentiation(akabackpropagation)Distributedrepresentations
• SimilaritiesanddifferencesbetweenGMsandNNs• Graphicalmodelsvs.computationalgraphs• SigmoidBeliefNetworksasgraphicalmodels• DeepBeliefNetworksandBoltzmannMachines
• CombiningDLmethodsandGMs• UsingoutputsofNNsasinputstoGMs• GMswithpotentialfunctionsrepresentedbyNNs• NNswithstructuredoutputs
History- Motivation
3
PerceptronandNeuralNetworks
4
ThePerceptronLearningAlgorithm
5
ThePerceptronLearningAlgorithm
6
NeuralNetworkModel
7
Combinedlogisticmodels
8
Combinedlogisticmodels
9
Combinedlogisticmodels
10
Notreally,notargetforhiddenunits...
11
Backpropagation:Reverse-modedifferentiation
12
Backpropagation:Reverse-modedifferentiation
13
Modelbuildingblocks
14
Modelbuildingblocks
15
Buildingblocksofdeepnetworks
16
Buildingblocksofdeepnetworks
17
Hand-craftedfeatures
18
Hand-craftedfeatures
19
UsingDNNsforhierarchicalrepresentations
20
21
GraphicalmodelsvsDeepnets
22
GraphicalmodelsvsDeepnets
23
GraphicalmodelsvsDeepnets
24
GraphicalmodelsvsDeepnets
25
GraphicalmodelsvsDeepnets
26
RestrictedBoltzmannMachines
27
RestrictedBoltzmannMachines:LearningandInference
28
RestrictedBoltzmannMachines:LearningandInference
29
RestrictedBoltzmannMachines:LearningandInference
30
SigmoidBeliefNetworks
31
RBMsareinfinitebeliefnetworks
32
RBMsareinfinitebeliefnetworks
33
RBMsareinfinitebeliefnetworks
34
RBMsareinfinitebeliefnetworks
35
Deepbeliefnetworks:layer-wisepre-training
36
DeepBoltzmannMachines
37
DeepBoltzmannMachines
38
39
GraphicalmodelsvsDeepnets
40
CombiningsequentialNNsandGMs[Gravesetal.2013]
41
CombiningsequentialNNsandGMs[Gravesetal.2013]
42
HybridNNsandconditionalGMs
43
HybridNNsandconditionalGMs
44
HybridNNsandconditionalGMs
45
Dealingwithstructuredprediction[Domke 2012]
Summary
46
• DL&GM:thefieldsaresimilarinthebeginning(structure,energy,etc.),andthendivergetotheirownsignaturepipelines• DL:mosteffortisdirectedtocomparingdifferentarchitecturesandtheircomponents(basedonempiricalperformanceonadownstreamtask)• DLmodelsaregoodatlearningrobusthierarchicalrepresentationsfromthedataandsuitableforsimplereasoning(“low-levelcognition”)
• GM:lotsofeffortsaredirectedtoimprovinginferenceaccuracyandconvergencespeed• GMsarebestforprovablycorrectinferenceandsuitableforhigh-levelcomplexreasoningtasks(“high-levelcognition”)
• Convergenceofbothfieldsisverypromising!
