Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4...
-
Upload
abel-foster -
Category
Documents
-
view
216 -
download
0
description
Transcript of Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4...
![Page 1: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/1.jpg)
Quiz 1 review
![Page 2: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/2.jpg)
Evaluating Classifiers
Reading: T. Fawcett paper, link on class website,
Sections 1-4
Optional reading: Davis and Goadrich paper, link on class
website
![Page 3: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/3.jpg)
“Confusion matrix” for a given class c
Actual Predicted (or “classified”) True
False(in class c)
(not in class c)
True (in class c) TruePositiveFalseNegative
False (not in class c) FalsePositive TrueNegative
Evaluating classification algorithms
![Page 4: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/4.jpg)
Evaluating classification algorithms
• Accuracy: Fraction of correct answers out of all problems
• Precision: Fraction of true positives out of all predicted positives:
• Recall: Fraction of true positives out of all actual positives:
€
P =TP
TP + FP
€
R =TP
TP + FN
![Page 5: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/5.jpg)
Trading off precision against recall
.
.
.
w1
w2
w64
o
w0
+1x1
x2
x64
€
class(x) = y(x) = sgn(w0 +w1x1 +w2x2 + ...+wnxn )
where sgn(z) =−1 if y < 0 0 if y = 0
+1 if y > 0
⎧ ⎨ ⎪
⎩ ⎪
How can we improve precision (at the expenseof recall) with a fixed classifier?
€
P =TP
TP +FP
€
R =TP
TP + FN
![Page 6: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/6.jpg)
True False(“8”) (“0”)
True (“8”) 40 10
False (“0”) 30 120
True False(“8”) (“0”)
True (“8”) ? ?
False (“0”) ? ?
Old, with threshold of 0
New, with threshold of:
-∞
Example 1: Assume 200 sample digits, of which 50 have class “8”
Precision? Recall?
Actual
Actual
Predicted
Predicted
€
P =TP
TP + FP
€
R =TP
TP + FN
![Page 7: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/7.jpg)
True False(“8”) (“0”)
True (“8”) 40 10
False (“0”) 30 120
True False(“8”) (“0”)
True (“8”) ? ?
False (“0”) ? ?
Old, with threshold of 0
New, with threshold of:
+∞
Precision? Recall?
Actual
Actual
Predicted
Predicted
Example 2: Assume 200 sample digits, of which 50 have class “8”
€
P =TP
TP + FP
€
R =TP
TP + FN
![Page 8: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/8.jpg)
€
P =TP
TP + FP
€
R =TP
TP + FN
Results of classifier
Threshold Accuracy Precision Recall
.9
.8
.7
.6
.5
.4
.3
.2
.1
-∞
Creating a Precision/Recall Curve
![Page 9: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/9.jpg)
9
(“sensitivity”)(1 “specificity”)
![Page 10: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/10.jpg)
10
![Page 11: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/11.jpg)
€
True Positive Rate (= Recall) =TP
TP + FN
€
False Positive Rate =FP
TN + FP
Results of classifier
Threshold Accuracy TPR FPR
.9
.8
.7
.6
.5
.4
.3
.2
.1
-∞
Creating a ROC Curve
![Page 12: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/12.jpg)
12
Precision/Recall versus ROC curves
http://blog.crowdflower.com/2008/06/aggregate-turker-judgments-threshold-calibration/
![Page 13: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/13.jpg)
13
![Page 14: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/14.jpg)
14
![Page 15: Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.](https://reader035.fdocuments.us/reader035/viewer/2022070605/5a4d1aef7f8b9ab05997d39a/html5/thumbnails/15.jpg)
15