ROC & AUC, LIFT

רוזנפלד" אבי ר ד

Introduction to ROC curves

• ROC = Receiver Operating Characteristic

• Started in electronic signal detection theory (1940s - 1950s)

• Has become very popular in biomedical applications, particularly radiology and imaging

• מידע בכריית בשימוש גם

False Positives / Negatives

P N

P 20 10

N 30 90

Predicted

Actu

al

Confusion matrix 1

P N

P 10 20

N 15 105

Predicted

Actu

al

Confusion matrix 2

FN

FP

Precision (P) = 20 / 50 = 0.4Recall (P) = 20 / 30 = 0.666F-measure=2*.4*.666/1.0666=.5

4

Different Cost Measures• The confusion matrix (easily generalize to multi-class)

• Machine Learning methods usually minimize FP+FN • TPR (True Positive Rate): TP / (TP + FN) = Recall• FPR (False Positive Rate): FP / (TN + FP) = Precision

Predicted class

Yes No

Actual class

Yes TP: True positive

FN: False negative

No FP: False positive

TN: True negative

Specific Example

Test Result

People with disease

People without disease

Test Result

Call these patients “negative”

Call these patients “positive”

Threshold

Test Result

Call these patients “negative”

Call these patients “positive”

without the diseasewith the disease

True Positives

Some definitions ...

Test Result

Call these patients “negative”

Call these patients “positive”

without the diseasewith the disease

False Positives

Test Result

Call these patients “negative”

Call these patients “positive”

without the diseasewith the disease

True negatives

Test Result

Call these patients “negative”

Call these patients “positive”

without the diseasewith the disease

False negatives

Test Result

without the diseasewith the disease

‘‘-’’

‘‘+’’

Moving the Threshold: left

Which line has the higher recall of -?Which line has the higher precision of -?

Tru

e P

osi

tive R

ate

(R

eca

ll)

0%

100%

False Positive Rate (1-specificity)

0%

100%

ROC curve

Figure 5.2 A sample ROC curve.

של שונים גרפים ROCסוגים

Area under ROC curve (AUC)

כללי • מדד

לגרף • מתחת ROCהשטח

•0.50 , רנדומאלי מחירה .1.0הוא מושלם הוא

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0%

100%

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0%

100%

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0%

100%

AUC = 50%

AUC = 90% AUC =

65%

AUC = 100%

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0%

100%

AUC for ROC curves

Lift Charts

• X axis is sample size: (TP+FP) / N• Y axis is TP• / : רנדומאלי דיוק המודל דיוק פורמאלי הגדרה

40% of responses for 10% of costLift factor = 4

80% of responses for 40% of costLift factor = 2Model

Random

Lift factor

0

0.5

1

1.5

2

2.5

3

3.5

4

4.55 15 25 35 45 55 65 75 85 95

Lift

Sample Size

Lift

Val

ue

המדדים בין הקשר

ה OVERFITTINGבעיית

10-fold cross-validation (one example of K-fold cross-validation)

• 1. Randomly divide your data into 10 pieces, 1 through k.• 2. Treat the 1st tenth of the data as the test dataset. Fit

the model to the other nine-tenths of the data (which are now the training data).

• 3. Apply the model to the test data (e.g., for logistic regression, calculate predicted probabilities of the test observations).

• 4. Repeat this procedure for all 10 tenths of the data.• 5. Calculate statistics of model accuracy and fit (e.g., ROC

curves) from the test data only.

תמונה

התוצאות ניתוח

The Kappa Statistic

• Kappa measures relative improvement over random prediction• Dreal / Dperfect = A (accuracy of the real model)

• Drandom / Dperfect= C (accuracy of a random model)• Kappa Statistic = (A-C) / (1-C)= (Dreal / Dperfect – Drandom / Dperfect ) / (1 – Drandom / Dperfect )

Remove Dperfect from all places

• (Dreal – Drandom) / (Dperfect – Drandom) • Kappa = 1 when A = 1• Kappa 0 if prediction is no better than random guessing

Aside: the Kappa statistic• Two confusion matrix for a 3-class problem: real model (left) vs

random model (right)

• Number of successes: sum of values in diagonal (D)• Kappa = (Dreal – Drandom) / (Dperfect – Drandom)

– (140 – 82) / (200 – 82) = 0.492– Accuracy = 140/200 = 0.70

a b c

a 88 10 2 100

b 14 40 6 60

c 18 10 12 40

120

60 20 200

Actu

al

Predicted

total

total a b c

a 60 30 10 100

b 36 18 6 60

c 24 12 4 40

120

60 20 200

Actu

al

Predicted

total

total

The kappa statistic – how to calculate Drandom ?

a b c

a 88 10 2 100

b 14 40 6 60

c 18 10 12 40

120

60 20 200

Actu

al

total

total a b c

a ? 100

b 60

c 40

120

60 20 200

Actu

altotal

total

100*120/200 = 60Rationale: 100 actual values, 120/200 in the predicted class, so random is:100*120/200

Actual confusion matrix, C

Expected confusion matrix, E, for a random model

התרגיל ...לקראת