RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for...
Transcript of RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for...
![Page 1: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/1.jpg)
RESTRICTED BOLTZMANN MACHINESDANIEL KOHLSDORF
![Page 2: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/2.jpg)
LAST LECTURE: DEEP AUTO ENCODERS
Directed Model
Reconstructs the input
Back propagation
Today:
Probabilistic Interpretation
Undirected Model
![Page 3: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/3.jpg)
DIRECTED VS UNDIRECTED MODELS
VS
![Page 4: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/4.jpg)
PROBABILISTIC UNDIRECTED MODELS
![Page 5: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/5.jpg)
PRELIMINARIES: MARKOV RANDOM FIELD
CliquesProbability Distribution
![Page 6: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/6.jpg)
RESTRICTED BOLTZMANN MACHINE
Hinton: A Practical Guide to Training Restricted Boltzmann Machines
![Page 7: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/7.jpg)
GIBBS SAMPLING
![Page 8: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/8.jpg)
GIBBS SAMPLING FOR RBM
h0 ~ p(h0 | v0,v1,v2,v3, h1,h2)h1 ~ p(h1 | v0,v1,v2,v3, h0,h2)h2 ~ p(h2 | v0,v1,v2,v3, h1,h0)h0, h1, h2 are independent
![Page 9: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/9.jpg)
GIBBS SAMPLING FOR RBM
h0 ~ p(h0 | v0,v1,v2,v3)h1 ~ p(h1 | v0,v1,v2,v3)h2 ~ p(h2 | v0,v1,v2,v3)
![Page 10: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/10.jpg)
RBM HIDDEN CONDITIONAL
p(h0 | v0,v1,v2,v3)
![Page 11: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/11.jpg)
RBM HIDDEN CONDITIONAL
h0 = p(h0 = 1 | v0,v1,v2,v3) > Uniform(0, 1)h1 = p(h1 = 1 | v0,v1,v2,v3) > Uniform(0, 1)h2 = p(h2 = 1 | v0,v1,v2,v3) > Uniform(0, 1)
![Page 12: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/12.jpg)
RBM VISIBLE CONDITIONAL
v0 = p(v0 = 1 | h0,h1,h2) > Uniform(0, 1)v1 = p(v1 = 1 | h0,h1,h2) > Uniform(0, 1)v2 = p(v2 = 1 | h0,h1,h2) > Uniform(0, 1)v3 = p(v3 = 1 | h0,h1,h2) > Uniform(0, 1)
![Page 13: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/13.jpg)
ALTERNATE GIBBS SAMPLING
h0, h1, h2
v0, v1, v2, v3
h0, h1, h2
v0, v1, v2, v3 …
![Page 14: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/14.jpg)
LEARNINGh0, h1, h2
v0, v1, v2, v3
h0, h1, h2
v0, v1, v2, v3…
Input Reconstruction
Weight Update
![Page 15: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/15.jpg)
LEARNING: CONTRASTIVE DIVERGENCE
h0, h1, h2
v0, v1, v2, v3 v0, v1, v2, v3
Input Reconstruction
Just do it once!
![Page 16: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/16.jpg)
DEEP!Hinton, G. E. and Salakhutdinov, R. R. (2006) Reducing the dimensionality of data with neural networks. Science
![Page 17: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/17.jpg)
CONVOLUTIONAL RESTRICTED BOLTZMANN MACHINES
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.
![Page 18: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/18.jpg)
Edge Detector Gaussian
From Aaron
![Page 19: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/19.jpg)
From Aaron
![Page 20: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/20.jpg)
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.
![Page 21: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/21.jpg)
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.
![Page 22: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/22.jpg)
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.
![Page 23: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/23.jpg)
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.
![Page 24: RESTRICTED BOLTZMANN MACHINESafb/classes/CS7616-Spring... · Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger](https://reader035.fdocuments.us/reader035/viewer/2022081613/5fbb7ceb2024a2522148e405/html5/thumbnails/24.jpg)
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.