Quadratic Classifiers(QC)
Quadratic Classifiers(QC)
J.-S. Roger Jang J.-S. Roger Jang ((張智星張智星 ))
CS Dept., National Taiwan Univ.CS Dept., National Taiwan Univ.http://mirlab.org/janghttp://mirlab.org/jang
[email protected]@mirlab.org
2010 Scientific Computing
2010 Scientific Computing
2
Bayes ClassifierBayes Classifier
Bayes classifier A probabilistic framework for classification problem
Conditional probability
Bayes theorem
112/04/18 2
)(/)()|(
)(/)()|(
iii
ii
CPCAPCAP
APACPACP
)(
)()|()|(
AP
CPCAPACP ii
i
2010 Scientific Computing
3 112/04/18 3
PDF ModelingPDF Modeling
• Goal:• Find a PDF (probability density function) that can
best describe a given dataset
• Steps:• Select a class of parameterized PDF• Identify the parameters via MLE (maximum
likelihood estimate) based on a given set of sample data
• Commonly used PDFs:• Multi-dimensional Gaussian PDF• Gaussian mixture models (GMM)
2010 Scientific Computing
4 112/04/18 4
PDF Modeling for ClassificationPDF Modeling for Classification
• Procedure for classification based on PDF• Training stage: PDF modeling of each class based
on the training dataset• Test stage: For each entry in the test dataset, pick
the class with the max. PDF
• Commonly used classifiers:• Quadratic classifier (n-dim. Gaussian PDF)• Gaussian-mixture-model classifier (GMM PDF)
2010 Scientific Computing
5 112/04/18 5
1D Gaussian PDF Modeling1D Gaussian PDF Modeling
• 1D Gaussian PDF:
• MLE of and •Detailed derivation
2
2
12
2
1),;(
x
exf
n
ii
n
ii
xn
xn
1
22
1
1
1
2010 Scientific Computing
6 112/04/18 6
1D Gaussian PDF Modeling via MLE1D Gaussian PDF Modeling via MLE
• MLE: Maximum Likelihood Estimate•Given a set of observations, find the parameters of the PDF such that the overall likelihood is maximized.
• Detailed derivation
2 2
2 21 2
1
1
22
1
; , ; ,
, ; , ; ,
1
1
i i i
n
n ii
n
ii
n
ii
x x p x g x
X x x x p X g x
xn
xn
2010 Scientific Computing
7 112/04/18 7
1D Gaussian PDF Modeling via MLE1D Gaussian PDF Modeling via MLE
Normal dist.estimated bynormal dist.
Uniform dist.estimated bynormal dist.
2010 Scientific Computing
8 112/04/18 8
d-dim. Gaussian PDF Modelingd-dim. Gaussian PDF Modeling
• d-dim. Gaussian PDF g(x, , )
• MLE of and :•Detailed derivation
)()(2
1
2/12/
1
)2(
1),;(
xx
d
T
exg
n
i
Tii
n
ii
xxn
xn
1
1
1
1
2010 Scientific Computing
9 112/04/18 9
d-dim. Gaussian PDF Modelingd-dim. Gaussian PDF Modeling
• d-dim. Gaussian PDF g(x, , )
• Likelihood of x in class j (governed by g(x, j, j))
)()(2
1
2/12/
1
)2(
1),;(
xx
d
T
exg
)()(2
1
2/12/
1
)2(
1),;(
jjT
j xx
jd
jj exg
2010 Scientific Computing
10 112/04/18 10
2D Gaussian PDF2D Gaussian PDF
Bivariate normal density:
Density function Contours
25.0
5.03,
0
0
2010 Scientific Computing
11
2D Gaussian PDF Modeling2D Gaussian PDF Modeling
gaussianMle.m
112/04/18 11
-2 -1 0 1 2
-1
0
1
-20
2
-1.5
-1
-0.5
0
0.5
1
1.5
0.1
0.2
0.3
0.4
2010 Scientific Computing
12
Steps of QCSteps of QC
Training stage• Select a type of Gaussian PDF• Identify the PDF of each class
Test stage• Assign each sample to the class with the highest
PDF value
112/04/18 12
2010 Scientific Computing
13
Characteristics of QCCharacteristics of QC
If each class is modeled by an Gaussian PDF, the decision boundary between any two classes is a quadratic function.• That is why it is called quadratic classifier.• How to prove it?
Different selections of the covariance matrix:• Constant times an identity matrix• Diagonal matrix• Full matrix (hard to use if the input dimension is
large)
112/04/18 13
2010 Scientific Computing
14
QC Results on Iris Dataset (I)QC Results on Iris Dataset (I)
Dataset: IRIS dataset with the last two inputs
112/04/18 14
2010 Scientific Computing
15
QC Results on Iris Dataset(II)QC Results on Iris Dataset(II)
PDF for each class:
112/04/18 15
2040
605
10152025
2
4
PDF of class 1
2040
605
10152025
0.20.40.60.8
11.2
PDF of class 2
2040
605
10152025
0.10.20.30.40.5
PDF of class 3
20 40 60
5
10
15
20
25
20 40 60
5
10
15
20
25
20 40 60
5
10
15
20
25
2010 Scientific Computing
16
QC Results on Iris Dataset (III)QC Results on Iris Dataset (III)
Decision boundaries among classes:
112/04/18 16
10 20 30 40 50 60
5
10
15
20
253 error points denoted by "x".
sepal length
sepa
l wid
th
44
Top Related