Unit 5

Probabilistic Model

• Machine Learning Task

Given X and Y

X is Feature Vector

Y is target value associated with X

• For given training data as (X,Y) pair the objective is:

i. To construct a learning Model

ii. To use the learning model to predict value of Y associated with unseen X

• X can be x1,x2… xi Y can be y1,y2…yi

• For binary Classification problem j=2 therefor y = y1 and y2

• Let N number of training pairs, (X,Y) are given as i/p.

• Yj= all feature vectors or instance belongs to Yj

• Rj = number of feature vectors belongs to class Yj

• Xi= set of feature value or instances of type Xi

• Ci= total number of instances of type xi

• Nij= number of features of type Xi belonging class Yj

xi

Yj

Ci

rj

Joint Probability Example

P(Red and Ace)

Black

Color Type Red Total

Ace 2 2 4

Non-Ace 24 24 48

Total 26 26 52

52

2

cards of number total

ace and red are that cards of number

Join Probability

• What is the probability that selected sample of type Xi and also belongs to class Yj

P(X=xi, Y=yj)

= P(X=xi ^ Y=yj)

=

Marginal Probability Example

P(Ace)

Black

Color Type Red Total

Ace 2 2 4

Non-Ace 24 24 48

Total 26 26 52

52

4

52

2

52

2)BlackandAce(P)dReandAce(P

Marginal Probability

• Let an instance is selected from input data

• What is the probability that selected sample is of type Xi

• Required Probability

• P(X=xi)=

Conditional Probabilities

Liberal Moderate Conservative Total

Gender Male 17 29 14 60

Female 30 24 23 77

Total 47 53 37 137

Political views

8

Here is a contingency table that gives the counts of ECO 138 students by their gender and political views. (Data are from Fall 2005 Class Survey)

P(Female) = 77/137 = 0.562

P(Female and Liberal) = 30/137 = 0.219

What is the probability that a selected student has moderate political views given that we have selected a female?

Conditional Probabilities (continued)

Liberal Moderate Conservative Total

Gender Male 17 29 14 60

Female 30 24 23 77

Total 47 53 37 137

Political views

9

What is the probability that a selected student has moderate political views given that we have selected a female?

P(Moderate | Female) = 24/77 = 0.311

Conditional probability, P (B|A) – the probability of event B given event A.

Conditional Probabilities (continued)

10

Formal Definition:

P(B|A) = P(A and B)

P(A)

Example: P(Moderate and Female)

P(Female)

=(24/137) / (77/137)

= 0.175 / 0.562

= 0.311

Conditional Probability

• Let an instance is selected from input data

• What is the probability that selected sample is of type Xi

• What is the probability that selected input belongs to Yj

• P(Y=yj | X= xi) =

• Conditional probability:

• Bayes theorem:

Bayesian classification

)(

)()|()|(

x

xx

p

CpCpCp

)(

),()|(

)(

),()|(

Cp

CpCp

p

CpCp

xx

x

xx

posterior

probability

likelihood prior probability

evidence

Naïve Bayes Classification

• Based on Bayes Rule

Finally, we classify X as RED since its class membership achieves the largest posterior probability.

Naïve Bayes Solved Example

OUTLOOK TEMPERATURE HUMIDITY WINDY PLAY GOLF

0 Rainy Hot High False No

1 Rainy Hot High True No

2 Overcast Hot High False Yes

3 Sunny Mild High False Yes

4 Sunny Cool Normal False Yes

5 Sunny Cool Normal True No

6 Overcast Cool Normal True Yes

7 Rainy Mild High False No

8 Rainy Cool Normal False Yes

9 Sunny Mild Normal False Yes

10 Rainy Mild Normal True Yes

11 Overcast Mild High True Yes

12 Overcast Hot Normal False Yes

13 Sunny Mild High True No

Example

In this example we have 4 inputs (predictors). The final posterior probabilities can be standardized between 0 and 1.

P (N0 | Today) > P (Yes | Today)

So, prediction that golf would be played is ‘No’.