Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning...
Transcript of Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning...
![Page 1: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/1.jpg)
Machine Learning Overview
Lyle Ungar
![Page 2: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/2.jpg)
Kinds of machine learningu Supervisedu Unsupervisedu Semi-supervisedu Reinforcementu Flavors
l Regression vs. classificationl Parametric vs. nonparametricl Active vs. passivel Single task vs. multi-task
![Page 3: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/3.jpg)
Supervised learningu Non-parametricu Parametric
l Minimize errorl Maximize likelihood (MLE/MAP)
u ‘Semiparametric’
Bias-Variance tradeoff
![Page 4: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/4.jpg)
Supervised learningu Non-parametric
l K-NN, Decision Trees, Random Forests, Boosted Trees
u Parametric l Regression: linear, logistic, LMSl Large margin: SVM, perceptron
u Semiparametricl neural nets
![Page 5: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/5.jpg)
Loss functionsu Real y
l L2
l L1
u Categorical yl L0
l Hingel Log loss: - Si log(pi)
n pi = the estimated probability of the correct answern minimizes KL(y|p)
![Page 6: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/6.jpg)
Which loss function for classification?u L2 vs log loss
l Which is preferred? Why?
u L0 vs hinge vs log lossl Which is most “hard”?l Which is most “soft”?l Which fits a probability model?
![Page 7: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/7.jpg)
Unsupervised learningu Projection vs. clusteringu Minimize reconstruction error
l PCAl K-meansl Auto-encoders
u Maximize likelihood l Gaussian Mixture Model (GMM)l LDAl Belief nets, including Naïve Bayes
![Page 8: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/8.jpg)
When to mean center for PCA?u Product purchases (e.g. amazon)u Word counts (e.g. twitter)u Pixels (e.g. brain scans)
xx
xx x
x xx
x1
x1
x2 x2
A B
![Page 9: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/9.jpg)
When (not) to rescaleu OLSu Ridge, elastic netu K-NNu RBFu PCRu SVMu Convolutional neural netu Random forest, boosted trees
Scale invariant?
![Page 10: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/10.jpg)
Method Selection: How big is n vs p ?u p >> nu n >> pu n ~ p
![Page 11: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/11.jpg)
Method Selection: How big is n vs p ?u p >> n: use dimensionality reduction
l or do extreme feature selection (RIC)l Then often just fit a linear modell Try semi-supervised learning
u n >> p: fit a flexible modell random forest, NNet, boosted treesl or look for more features
u n ~ p: consider feature selection – and dim. reductionl Elastic net?
![Page 12: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/12.jpg)
What do you know about your problem?u Are features highly correlated or almost independent?u Roughly linear or highly nonlinear?u Is noise Gaussian? u Conditional independence or causal structure?u Constraints?u Fixed size or variable length feature set?u What is your real loss function?
![Page 13: Machine Learning Overview - Penn Engineeringcis520/lectures/overview.pdf · Machine Learning Overview Lyle Ungar. Kinds of machine learning uSupervised uUnsupervised uSemi-supervised](https://reader036.fdocuments.us/reader036/viewer/2022062415/5fbeb11d1ba2b130397a53fb/html5/thumbnails/13.jpg)
What method to use? Why?Data #y classes n p
u MRI 2 100 10,000u Image 1,000 500,000 600u Disease 3 1,000 50u Disease 10 1,000 200u Text in docs 2 40,000 40,000u Student apps 2 5,000 500