Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning...
Transcript of Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning...
![Page 1: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/1.jpg)
Introduction to Deep LearningPrinceton University COS 495
Instructor: Yingyu Liang
![Page 2: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/2.jpg)
What is deep learning?
• Short answer: recent buzz word
![Page 3: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/3.jpg)
Industry
• Microsoft
• …
• Musk
• Toyota
• Drug
• Finance
![Page 4: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/4.jpg)
Industry
![Page 5: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/5.jpg)
Industry
![Page 6: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/6.jpg)
Industry
• Microsoft
![Page 7: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/7.jpg)
Industry
• Elon Musk
![Page 8: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/8.jpg)
Industry
• Toyota
![Page 9: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/9.jpg)
Academy
• NIPS 2015: ~4000 attendees, double the number of NIPS 2014
![Page 10: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/10.jpg)
Academy
• Science special issue
• Nature invited review
![Page 11: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/11.jpg)
What is deep learning?
• Longer answer: machine learning framework that shows impressive performance on many Artificial Intelligence tasks
![Page 12: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/12.jpg)
Image
• Image classification• 1000 classes
Slides from Kaimin He, MSRA
Human performance: ~5%
![Page 13: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/13.jpg)
Image
• Object location
Slides from Kaimin He, MSRA
![Page 14: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/14.jpg)
Image
• Image captioning
Figure from the paper “DenseCap: Fully Convolutional Localization Networks for Dense Captioning”, by Justin Johnson, Andrej Karpathy, Li Fei-Fei
![Page 15: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/15.jpg)
Text
• Question & Answer
Figures from the paper “Ask Me Anything: Dynamic Memory Networks for Natural Language Processing ”,by Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Richard Socher
![Page 16: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/16.jpg)
Game
Google DeepMind's Deep Q-learning playing Atari BreakoutFrom the paper “Playing Atari with Deep Reinforcement Learning”,by Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou,Daan Wierstra, Martin Riedmiller
![Page 17: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/17.jpg)
Game
![Page 18: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/18.jpg)
The impact
• Revival of Artificial Intelligence
• Next technology revolution?
• A big thing ongoing, should not miss
![Page 19: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/19.jpg)
Questions behind the scene
• Return of artificial neural network• What’s different
• Why get great performance
• Future development• The road to general-purpose AI?
![Page 20: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/20.jpg)
Goal of the course
• Introduction
• Key concepts
• Ticket to the party
![Page 21: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/21.jpg)
Syllabus
• Part I: machine learning basics• Linear model, Perceptron, SVM
• Multi-class
• Training by gradient descent
• overfitting
• Part II: supervised deep learning (feedforward network)
• Part III: unsupervised learning
• Part IV: deep learning in the wild
![Page 22: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/22.jpg)
Syllabus
• Part I: machine learning basics
• Part II: supervised deep learning (feedforward network)• Multiple-layer and Backpropogation
• Regularization
• Convolution
• Part III: unsupervised deep learning
• Part IV: deep learning in the wild
![Page 23: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/23.jpg)
Syllabus
• Part I: machine learning basics
• Part II: supervised deep learning (feedforward network)
• Part III: unsupervised deep learning• PCA
• Boltzmann machine, Deep Boltzmann machine
• autoencoder
• Part IV: deep learning in the wild
![Page 24: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/24.jpg)
Syllabus
• Part I: machine learning basics
• Part II: supervised deep learning (feedforward network)
• Part III: unsupervised deep learning
• Part IV: deep learning in the wild• Read papers on advanced topics
• Play with the code
• Presentation
![Page 25: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/25.jpg)
Textbook and materials
• Deep Learning:
http://www.deeplearningbook.org/
• Suggested software framework: Tensorflow• in Python
• Easy to install/use
• Can try it on your laptop
• Other software frameworks: Theano, Caffe, Torch, Marvin, …
![Page 26: Princeton University COS 495 Instructor: Yingyu Liang...Game Google DeepMind's Deep Q-learning playing Atari Breakout From the paper “Playing Atari with Deep Reinforcement Learning”,](https://reader030.fdocuments.us/reader030/viewer/2022040204/5ec9714ec18133726b4d566f/html5/thumbnails/26.jpg)
Grading
• Problem Sets (5 sets): 70%
• Design Projects: 25%
• Oral Presentation: 5%