Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine...
Transcript of Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine...
![Page 1: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/1.jpg)
Machine Learning algorithms and methods in Weka
Presented by: William ElazmehPhD. candidate at the Ottawa-Carleton Institute for Computer
Science, University of Ottawa, Canada
Abstract:this workshop presents a review of concepts and methods used in machine learning. The workshop aims to illustrate such ideas using the Weka software. The workshop is divided into 3 parts;
(1) an illustration of data processing and using machine learning algorithms in Weka, (2) a demonstration of experiment administrations in Weka, and (3) a talk on evaluating machine
learning algorithms using ROC and Cost Curves.
![Page 2: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/2.jpg)
•Machine learning/data mining software written in Java (distributed under the GNU Public License)•Used for research, education, and applications•Complements “Data Mining” by Witten & Frank
Main features:•Data pre-processing tools•Learning algorithms•evaluation methods
•Graphical user interfaces •An environment experimenting
WEKA is a Machine Learning Toolkit that consists of:•The Explorer
•Classification and Regression•Clustering•Finding Associations•Attribute Selection•Data Visualization
•The Experimenter•The Knowledge Flow GUI
Note: the content of this presentation is based on a Weka presentation prepared by EibeFrank at the Department of Computer Science, University of Waikato, New Zealand
![Page 3: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/3.jpg)
The ARFF Flat File Format
class
Nominal attribute
Numeric attribute
Data records
![Page 4: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/4.jpg)
Command line console
![Page 5: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/5.jpg)
Weka Explorer
![Page 6: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/6.jpg)
Weka Experimenter
![Page 7: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/7.jpg)
Weka Knowledge Flow Designer
![Page 8: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/8.jpg)
Weka ARFF Viewer
![Page 9: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/9.jpg)
Weka Log Viewer
![Page 10: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/10.jpg)
Part I: data processing and using machine learning algorithms in
Weka
![Page 11: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/11.jpg)
Explorer: pre-processing the data
• Data can be imported from a file in various formats: ARFF, CSV, C4.5, binary
• Data can also be read from a URL or from an SQL database (using JDBC)
• Pre-processing tools in WEKA are called “filters”
• WEKA contains filters for:Discretization, normalization, resampling, attribute
selection, transforming and combining attributes, …
![Page 12: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/12.jpg)
Preprocessing
![Page 13: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/13.jpg)
Data Visualization
![Page 14: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/14.jpg)
Data Filtering
![Page 15: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/15.jpg)
![Page 16: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/16.jpg)
![Page 17: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/17.jpg)
Explorer: building “classifiers”
• Classifiers in WEKA are models for predicting nominal or numeric quantities
• Implemented learning schemes include:– Decision trees and lists, instance-based
classifiers, support vector machines, multi-layer perceptrons, logistic regression, Bayes’ nets, …
• “Meta”-classifiers include:– Bagging, boosting, stacking, error-correcting
output codes, locally weighted learning, …
![Page 18: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/18.jpg)
Classification
![Page 19: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/19.jpg)
![Page 20: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/20.jpg)
![Page 21: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/21.jpg)
![Page 22: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/22.jpg)
![Page 23: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/23.jpg)
![Page 24: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/24.jpg)
![Page 25: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/25.jpg)
![Page 26: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/26.jpg)
![Page 27: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/27.jpg)
Explorer: clustering data
• WEKA contains “clusterers” for finding groups of similar instances in a dataset
• Implemented schemes are:– k-Means, EM, Cobweb, X-means, FarthestFirst
• Clusters can be visualized and compared to “true” clusters (if given)
• Evaluation based on loglikelihood if clustering scheme produces a probability distribution
![Page 28: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/28.jpg)
Clustering
![Page 29: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/29.jpg)
![Page 30: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/30.jpg)
![Page 31: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/31.jpg)
Explorer: finding associations• WEKA contains an implementation of the Apriori
algorithm for learning association rules
– Works only with discrete data
• Can identify statistical dependencies between groups of attributes:
– milk, butter ⇒ bread, eggs (with confidence 0.9 and support 2000)
• Apriori can compute all rules that have a given minimum support and exceed a given confidence
![Page 32: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/32.jpg)
Finding Associations
![Page 33: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/33.jpg)
![Page 34: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/34.jpg)
![Page 35: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/35.jpg)
Explorer: attribute selection• Panel that can be used to investigate which (subsets
of) attributes are the most predictive ones
• Attribute selection methods contain two parts:
– A search method: best-first, forward selection, random, exhaustive, genetic algorithm, ranking
– An evaluation method: correlation-based, wrapper, information gain, chi-squared, …
• Very flexible: WEKA allows (almost) arbitrary combinations of these two
![Page 36: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/36.jpg)
Attribute Selection
![Page 37: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/37.jpg)
![Page 38: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/38.jpg)
![Page 39: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/39.jpg)
![Page 40: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/40.jpg)
Explorer: data visualization• Visualization very useful in practice: e.g. helps to
determine difficulty of the learning problem
• WEKA can visualize single attributes (1-d) and pairs of attributes (2-d)
– To do: rotating 3-d visualizations (Xgobi-style)
• Color-coded class values
• “Jitter” option to deal with nominal attributes (and to detect “hidden” data points)
• “Zoom-in” function
![Page 41: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/41.jpg)
Data Visualization
![Page 42: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/42.jpg)
![Page 43: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/43.jpg)
![Page 44: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/44.jpg)
Part II: experiment administrations in Weka
![Page 45: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/45.jpg)
Performing experiments• Experimenter makes it easy to compare the
performance of different learning schemes
• For classification and regression problems
• Results can be written into file or database
• Evaluation options: cross-validation, learning curve, hold-out
• Can also iterate over different parameter settings
• Significance-testing built in!
![Page 46: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/46.jpg)
Weka Experimenter
![Page 47: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/47.jpg)
![Page 48: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/48.jpg)
![Page 49: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/49.jpg)
![Page 50: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/50.jpg)
The Knowledge Flow GUI• New graphical user interface for WEKA
• Java-Beans-based interface for setting up and running machine learning experiments
• Data sources, classifiers, etc. are beans and can be connected graphically
• Data “flows” through components: e.g.,
“data source” -> “filter” -> “classifier” -> “evaluator”
• Layouts can be saved and loaded again later
![Page 51: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/51.jpg)
Weka Knowledge Flow Designer
![Page 52: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/52.jpg)
![Page 53: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/53.jpg)
![Page 54: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/54.jpg)
![Page 55: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/55.jpg)
![Page 56: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/56.jpg)
Finally!WEKA is available at
http://www.cs.waikato.ac.nz/ml/weka
Also has a list of projects based on WEKA
WEKA contributors:
Abdelaziz Mahoui, Alexander K. Seewald, Ashraf M. Kibriya, Bernhard Pfahringer , Brent Martin, Peter Flach, Eibe Frank ,Gabi Schmidberger,Ian H. Witten , J. Lindgren, Janice Boughton, Jason Wells, Len Trigg, Lucio de Souza Coelho, Malcolm Ware, Mark Hall ,Remco Bouckaert , Richard Kirkby, Shane Butler, Shane Legg, Stuart Inglis, Sylvain Roy, Tony Voyle, Xin Xu, Yong Wang, Zhihai Wang
![Page 57: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/57.jpg)
Part III: evaluating machine learning algorithms using ROC
and Cost Curves
![Page 58: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/58.jpg)
Introduction to ROC Curves (Drummond et. al.)
• The focus is on visualization of classifier’s performance
• ROC curves show the tradeoff between false positive
• and true positive rates
• We want to know when and by how much a classifier outperforms another
• The analysis is restricted to a two class classifier
![Page 59: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/59.jpg)
Introduction to Cost Curves (Drummond et al.)
• The focus is on visualization of classifier’s performance
• ROC curves show the tradeoff between false positive
• and true positive rates
• We want to know when and by how much a classifier outperforms another
• The analysis is restricted to a two class classifier
![Page 60: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/60.jpg)
Cost Curves (Drummond et al. 2004)
• Given a specific misclassification cost and class probabilities, what is the expected cost of classification?
• For what misclassification costs and class probabilities
• does a classifier outperform the trivial classifiers?
• For what misclassification costs and class probabilities
• does a classifier outperform another?
• What is the difference in performance between two classifiers?
• What is the average performance of several independent classifiers?
• What is the 90% confidence interval for a particular classifier’s performance?
• What is the significance of the difference between the performances of two classifiers?
![Page 61: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/61.jpg)
![Page 62: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/62.jpg)
ROC Space
![Page 63: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/63.jpg)
Generating ROC
curves
![Page 64: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/64.jpg)
ROC curves
![Page 65: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/65.jpg)
Cost Space
![Page 66: Machine Learning algorithms and methods in Wekastan/csi7163/weka_workshop_slides.pdf · Machine Learning algorithms and methods in Weka Presented by: William Elazmeh PhD. candidate](https://reader031.fdocuments.us/reader031/viewer/2022030501/5aad826f7f8b9a8f498e59cc/html5/thumbnails/66.jpg)
Cost Curves