Post on 10-Jan-2016
description
Relevant characteristics extraction from semantically
unstructured data
PhD title : Data mining in unstructured data
Daniel I. MORARIU, MSc
PhD Supervisor: Lucian N. VINŢANSibiu, 2006
Contents Prerequisites Correlation of the SVM kernel’s parameters
Polynomial kernel Gaussian kernel
Feature selection using Genetic Algorithms Chromosome encoding Genetic operators
Meta-classifier with SVM Non-adaptive method – Majority Vote Adaptive methods
Selection based on Euclidean distance Selection based on cosine
Initial data set scalability Choosing training and testing data sets Conclusions and further work
Prerequisites Reuters Database Processing
806791 total documents, 126 topics, 366 regions, 870 industry codes
Industry category selection – “system software” 7083 documents (4722 training /2361 testing) 19038 attributes (features) 24 classes (topics)
Data representation Binary Nominal Cornell SMART
Classifier using Support Vector Machine techniques
kernels
Correlation of the SVM kernel’s parameters
Polynomial kernel
Gaussian kernel
dxxdxxk '2)',(
Cn
xxxxk
2'
exp)',(
Polynomial kernel Commonly used kernel
d – degree of the kernel b – the offset
Our suggestion b = 2 * d
Polynomial kernel parameter’s correlation
ddk '2)',( xxxx
dbk xxxx ),(
Bias – Polynomial kernel
dbk xxxx ),(Influence of the bias - Nominal representation of input data
65
70
75
80
85
90
0 1 2 3 4 5 6 7 8 9 10
50
10
0
50
0
10
00
13
09
Values of the bias (b)
Acc
ura
cy (
%)
d=1
d=2
d=3
d=4
OurChoice
ddk '2)',( xxxx
Gaussian kernel parameter’s correlation
Gaussian kernel Commonly used kernel
C – usually represents the dimension of the set
Our suggestion n – numbers of distinct features greater
than 0
Ck
'exp)',(
xxxx
Cn
k2
'exp)',(
xxxx
n – Gaussian kernel
Ck
'exp)',(
xxxx
Influence of n - Cornell Smart data representation
50
55
60
65
70
75
80
85
90
1 10 50 100 500 654 1000 1309 auto
Values of parameter n
Acc
urac
y (%
) C=1.0
C=1.3
C=1.8
C=2.1
Cn
k2
'exp)',(
xxxx
auto
Feature selection using Genetic Algorithms
Chromosome
Fitness (ci) = SVM (ci)
Methods of selecting parents Roulette Wheel Gaussian selection
Genetic operators Selection Mutation Crossover
bwwwc ,,...,, 1903810
m
iin bbwwwfcf
121 ,),,...,,(()( xw
Methods of selecting the parents
Roulette Wheel each individual is represented by a space
that corresponds proportionally to its fitness
Gaussian : maxim value (m=1) and dispersion (σ =
0.4)
2
)((
2
1exp)(
mcfitness
cP ii
The process of obtaining the next
generation
Current generation
The best chromosome is copied from old
population into the new population
Selects two parents.
We create two children from selected parents using crossover with parents split
Need more chromosomes into the set?
Randomly eliminate one of the parents
Mutation – randomly change the sign for a random number of
elements
Selection Crossover Mutation
New generation
Yes
No
GA_FS versus SVM_FS for 1309 features
0
10
20
30
40
50
60
70
80
90Acc
ura
cy(%
)
D1.0 D2.0 D3.0 D4.0 D5.0
Kernel degree
GA-BINGA-NOMGA-CSSVM-BINSVM-NOMSVM-CS
Training time, polynomial kernel, d= 2, NOM
0
10
20
30
40
50
60
70
80
475 1309 2488 8000
number of features
Tim
e[m
inute
s]
GA_FSSVM_FSIG_FS
GA_FS versus SVM_FS for 1309 features
81.5
82
82.5
83
83.5
84Acc
ura
cy(%
)
C1.0 C1.3 C1.8 C2.1 C2.8 C3.1
Parameter C
GA-BINGA-CSSVM-BINSVM-CS
Training time, Gaussian kernel, C=1.3, BIN
0
20
40
60
80
100
120
475 1309 2488 8000
number of features
Tim
e[m
inute
s]
GA_FSSVM_FSIG_FS
Meta-classifier with SVM
Set of SVM’s Polynomial degree 1, Nominal Polynomial degree 2, Binary Polynomial degree 2, Cornell Smart Polynomial degree 3, Cornell Smart Gaussian C=1.3, Binary Gaussian C=1.8, Cornell Smart Gaussian C=2.1, Cornell Smart Gaussian C=2.8, Cornell Smart
Upper limit (94.21%)
Meta-classifier methods’
Non-adaptive method Majority Vote – each classifier votes a
specific class for a current document Adaptive methods - Compute the similarity between a
current sample and error samples from the self queue
Selection based on Euclidean distance First good classifier The best classifier
Selection based on cosine First good classifier The best classifier Using average
n
ii
n
ii
n
iii
xx
xx
1
2
1
2
1
]'[][
]'[][
'
',cos
xx
xx
n
iii xxEucl
1
2)][]([),( xx
Selection based on Euclidean distance
Classification accuracy
78808284868890929496
1 3 5 7 9 11 13
Steps
Acc
ura
cy(%
)
Upper LimitFC-SBEDBC-SBED
Selection based on cosine
Classification accuracy
80
82
84
86
88
90
92
94
96
1 3 5 7 9 11 13
Steps
Acc
ura
cy(%
)
Upper Limit
FC-SBCOS
BC-SBCOS
BC-SBCOS - withaverage
Comparison between SBED and SBCOS
Classification Accuracy
80
82
84
86
88
90
92
94
96
1 3 5 7 9 11 13
Steps
Acc
ura
cy(%
)
Majority VoteSBEDSBCOSUpper Limit
Comparison between SBED and SBCOS
Processing Time
0
10
20
30
40
50
60
70
80
1 3 5 7 9 11 13
Steps
Tim
e [
min
ute
s]
Majority VoteSBEDSBCOS
Initial data set scalability
Normalize each sample (7053)Group initial set based on distance (4474)
Take relevant vector (4474)Use relevant vector in classification process
Select only support vectors (847)
Take samples grouped in selected support vectors (4256)Make the classification (with 4256 samples)
Polynomial kernel – 1309 features, NOM
74
76
78
80
82
84
86
88
Acc
ura
cy (
%)
D1.0 D2.0 D3.0 D4.0 D5.0
Degree of kernel
Kernel's degree influence
SVM -7053SVM-4256
Gaussian kernel – 1309 features, CS
70
72
74
76
78
80
82
84
86
88
90Acc
ura
cy(%
)
1 1.3 1.8 2.1 2.8
parameter C
SVM-7053SVM-4256
Training time
0
10
20
30
40
50
Tim
e [
min
ute
s]
C1.0 C1.3 C1.8 C2.1 C2.8
7053-Bin4256-Bin
Parameter C
7053-Bin7053-CS4256-Bin4256-CS
Choosing training and testing data set
74
76
78
80
82
84
86
88Acc
ura
cy(%
)
D1.0
D2.0
D3.0
D4.0
D5.0
Aver
age
Kernel's degree
1309 Features - Polynomial kernel
average over oldsetaverage over newset
Choosing training and testing data set
7072747678808284868890
Acc
ura
cy(%
)
C1.0
C1.3
C1.8
C2.1
C2.8
Aver
age
Kernel's degree
1309 Features - Gaussian kernel
average over oldsetaverage over newset
Conclusions – other results
Using our correlation 3% better for Polynomial kernel 15% better for Gaussian kernel
Reduced number of features between 2.5% (475) and 6% (1309)
GA _FS faster than SVM_FS Polynomial kernel with nominal representation
and small degree Gaussian kernel with Cornell Smart
representation Reuter’s database is linearly separable SBED is better and faster than SBCOS Classification accuracy decreases with 1.2 %
when the data set is reduced
Further work
Features extraction and selection
Association rules between words (Mutual Information)
Synonym and Polysemy problem Using families of words (WordNet)
Web mining application Classifying larger text data sets A better method of grouping data Using classification and clustering
together