Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 ›...
Transcript of Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 ›...
![Page 1: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/1.jpg)
Foundations of Machine Learning and Data Science
Maria-Florina (Nina) Balcan
Lecture 1, September 9, 2015
![Page 2: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/2.jpg)
Course Staff
• Nina Balcan http://www.cs.cmu.edu/~ninamf
• Avrim Blum http://www.cs.cmu.edu/~avrim
Instructors:
• Nika Haghtalab http://www.cs.cmu.edu/~nhaghtal
• Sarah Allen http://www.cs.cmu.edu/~srallen
TAs:
![Page 3: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/3.jpg)
Lectures in general
On the board
Ocasionally, will use slides
![Page 4: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/4.jpg)
4
Image Classification
Document Categorization
Speech Recognition
Branch Prediction
Protein Classification
Spam Detection Fraud Detection
Machine Learning
Playing Games Computational Advertising
![Page 5: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/5.jpg)
Machine Learning is Changing the World
“A breakthrough in machine learning would be worth ten Microsofts” (Bill Gates, Microsoft)
“Machine learning is the hot new thing” (John Hennessy, President, Stanford)
“Web rankings today are mostly a matter of machine learning” (Prabhakar Raghavan, VP Engineering at Google)
![Page 6: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/6.jpg)
The COOLEST TOPIC IN SCIENCE
• “A breakthrough in machine learning would be worth ten Microsofts” (Bill Gates, Chairman, Microsoft)
• “Machine learning is the next Internet” (Tony Tether, Director, DARPA)
• Machine learning is the hot new thing” (John Hennessy, President, Stanford)
• “Web rankings today are mostly a matter of machine learning” (Prabhakar Raghavan, Dir. Research, Yahoo)
• “Machine learning is going to result in a real revolution” (Greg Papadopoulos, CTO, Sun)
• “Machine learning is today’s discontinuity” (Jerry Yang, CEO, Yahoo)
![Page 7: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/7.jpg)
This course: foundations of Machine Learning
A2 Â
and Data Science
![Page 8: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/8.jpg)
• what kinds of tasks we can hope to learn, and from what kind of data
Goals of Machine Learning Theory Develop and analyze models to understand:
• what types of guarantees might we hope to achieve
• prove guarantees for practically successful algs (when will they succeed, how long will they take?)
• Algorithms
Interesting connections to other areas including:
• Optimization
• Probability & Statistics • Game Theory
• Information Theory • Complexity Theory
• develop new algs that provably meet desired criteria (potentially within new learning paradigms)
![Page 9: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/9.jpg)
9
Example: Supervised Classification
Goal: use emails seen so far to produce good prediction rule for future data.
Not spam spam
Decide which emails are spam and which are important.
Supervised classification
![Page 10: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/10.jpg)
10
example label
Reasonable RULES:
Predict SPAM if unknown AND (money OR pills)
Predict SPAM if 2money + 3pills –5 known > 0
Represent each message by features. (e.g., keywords, spelling, etc.)
Example: Supervised Classification
+
-
+ + +
- -
-
-
-
Linearly separable
![Page 11: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/11.jpg)
11
Two Main Aspects of Supervised Learning
Algorithm Design. How to optimize?
Automatically generate rules that do well on observed data.
Confidence Bounds, Generalization Guarantees, Sample Complexity
Confidence for rule effectiveness on future data.
Well understood for passive supervised learning.
![Page 12: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/12.jpg)
Using Unlabeled Data and Interaction for Learning
Computer Vision
Search/Information Retrieval
Computational Biology
Spam Detection
Medical Diagnosis Robotics
Application Areas
![Page 13: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/13.jpg)
13
Billions of webpages
Only a tiny fraction can be annotated by human experts.
Massive Amounts of Raw Data
Images Protein sequences
![Page 14: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/14.jpg)
14
Expert Labeler
Semi-Supervised Learning
raw data
face not face
Labeled data
Classifier
![Page 15: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/15.jpg)
15
Active Learning
face
O
O
O
Expert Labeler
raw data
Classifier
not face
![Page 16: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/16.jpg)
16
• Semi-Supervised Learning
Using cheap unlabeled data in addition to labeled data.
• Active Learning
The algorithm interactively asks for labels of informative examples.
Other Protocols for Supervised Learning
Theoretical understanding entirely lacking 10 years ago.
Lots of progress recently. We will cover some of these.
![Page 17: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/17.jpg)
Distributed Learning Many ML problems today involve massive amounts of data distributed across multiple locations.
Often would like low error hypothesis wrt the overall distrib.
![Page 18: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/18.jpg)
Distributed Learning
E.g., medical data
Data distributed across multiple locations.
![Page 19: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/19.jpg)
Distributed Learning
E.g., scientific data
Data distributed across multiple locations.
![Page 20: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/20.jpg)
Distributed Learning
• Data distributed across multiple locations.
• Each has a piece of the overall data pie.
Important question: how much communication?
Plus, privacy & incentives.
• To learn over the combined D, must communicate.
![Page 21: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/21.jpg)
The World is Changing Machine Learning
Many competing resources & constraints. E.g.,
• Computational efficiency (noise tolerant algos)
• Communication
• Human labeling effort
• Statistical efficiency
• Privacy/Incentives
New approaches. E.g.,
• Semi-supervised learning
• Distributed learning
• Interactive learning
• Multi-task/transfer learning
• Never ending learning
• Deep Learning
![Page 22: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/22.jpg)
Structure of the Class
• Simple algos and hardness results for supervised learning.
• Classic, state of the art algorithms: AdaBoost and SVM (kernel based mehtods).
• Basic models: PAC, SLT.
• Standard Sample Complexity Results (VC dimension)
• Weak-learning vs. Strong-learning
Basic Learning Paradigm: Passive Supervised Learning
• Modern Sample Complexity Results • Rademacher Complexity; localization
• Margin analysis of Boosting and SVM
![Page 23: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/23.jpg)
Structure of the Class
• Incorporating Unlabeled Data in the Learning Process.
• Incorporating Interaction in the Learning Process:
• Active Learning
• More general types of Interaction
Other Learning Paradigms
• Distributed Learning.
• Transfer learning/Multi-task learning/Life-long learning.
• Deep Learning.
• Foundations and algorithms for constraints/externalities. E.g., privacy, limited memory, and communication.
![Page 24: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/24.jpg)
Structure of the Class
• Online Learning, Optimization, and Game Theory
• connections to Boosting
Other Topics.
• Methods for summarizing and making sense of massive datasets including:
• unsupervised learning.
• spectral, combinatorial techniques.
• streaming algorithms.
![Page 25: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/25.jpg)
Admin
• Course web page:
http://www.cs.cmu.edu/~ninamf/courses/806/10-806-index.html
Two grading schemes:
1) Project Oriented.
- Project [60%]
- Take-home final [10%]
- Hwks + grading [30%]
2) Homework Oriented.
- Hwk +grading [60%]
- Take-home final [10%]
- Project [30%]
![Page 26: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/26.jpg)
Admin
1) Project Oriented.
- Project [60%]
- Take-home final [10%]
• explore a theoretical or empirical question; • write-up --- ideally aim for a conference submission! • Small groups OK.
- Hwks + grading [30%]
• Course web page:
http://www.cs.cmu.edu/~ninamf/courses/806/10-806-index.html
![Page 27: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/27.jpg)
Admin
2) Homework Oriented.
- Take-home final [10%]
- Project [30%]
• read a couple of papers and explain the idea.
- Hwk +grading [60%]
• Course web page:
http://www.cs.cmu.edu/~ninamf/courses/806/10-806-index.html
![Page 28: Foundations of Machine Learning and Data Science › ~ninamf › courses › 806 › lect09-09-slides.pdf · •Machine learning is the hot new thing” (John Hennessy, President,](https://reader035.fdocuments.us/reader035/viewer/2022081407/5f1d067bee69c75b3648ad03/html5/thumbnails/28.jpg)
Lectures in general
On the board
Ocasionally, will use slides