An SVM Based Voting Algorithm with Application to Parse Reranking
description
Transcript of An SVM Based Voting Algorithm with Application to Parse Reranking
![Page 1: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/1.jpg)
An SVM Based Voting Algorithm with Application to Parse Reranking
Paper by Libin Shen and Aravind K. JoshiPresented by Amit Wolfenfeld
![Page 2: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/2.jpg)
OutlineIntroduction of Parse RerankingSVMAn SVM Based Voting AlgorithmTheoretical JustificationExperiments on Parse RerankingConclusions
![Page 3: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/3.jpg)
Introduction – Parse RerankingMotivation (Collins)vote rerank f-
scoreLog-
likelihood
parses rank
3 92% -120.0 P2 1
4 90% -121.5 P3 2
x 1 96% -122.0 P1 3
2 93% -122.5 P4 4
![Page 4: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/4.jpg)
Support Vector MachinesThe SVM is a large margin
classifier that searches for the hyperplane that maximizes the margin between the positive samples and the negative samples
![Page 5: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/5.jpg)
Support Vector MachinesMeasures of the capacity of a
learning machine: VC Dimension, Fat Shattering Dimension
The capacity of a learning machine is related to the margin on the training data.- As the margin goes up, VC-dimension may go down and thus the upper bound of the test error goes down. (Vapnik 79)
![Page 6: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/6.jpg)
Support Vector MachinesSVMs’ theoretical accuracy is
much lower than their actual performance. The margin based upper bounds of the test error are too loose.
This is why – SVM based voting algorithm.
![Page 7: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/7.jpg)
SVM Based VotingPrevious work (Dijkstra 02)
- Use SVM for parse reranking directly.- Positive samples: parse with highest f-score for each sentence.
First try-Tree kernel: compute dot-product on the space of all the subtrees (Collins 02)-Linear kernel: rich features (Collins 00)
![Page 8: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/8.jpg)
SVM based Voting AlgorithmUsing pairwise parses as samplesLet is the j-th candidate parse
for the i-th sentence in the training data.
Let is the parse with highest f-score among all the parses for the i-th sentence.
Positive samples: Negative samples:
![Page 9: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/9.jpg)
Preference KernelsLet are two pairs of parses K – kernel : linear or tree kernelThe preference kernel is defined:
- +
A sample represents the difference between a good parse and a bad one, the preference computes the similarity between the two differences.
![Page 10: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/10.jpg)
SVM based Voting Decision function f of SVM:
for each of the pair parses:
is the i-th support vectoris the total number of support vectorsis the class of can be is the Lagrange multiplier solved by the SVM
![Page 11: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/11.jpg)
Theoretical IssuesJustifying the Preference Kernel
Justifying Pairwise Samples
Margin Based Bound for the SVM Based Voting Algorithm
![Page 12: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/12.jpg)
Justifying the Preference KernelThe kernelThe preference kernel
- - + - )(- )
![Page 13: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/13.jpg)
Justifying the Pairwise SamplesThe SVM using simple parses as
samples searches for a decision function score constrained by the condition:- - too strong.
Pairwise:-
![Page 14: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/14.jpg)
Margin Based Bound for SVM Based voting
Loss function of voting :
Loss function of classification:Expected voting loss is equal expected
classification loss(Herbrich 2000)
![Page 15: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/15.jpg)
Experiments – WSJ TreebankN-best parsing results (Collins 02)SVM-light (Joachims 98)Two Kernels (K) used in the preference
kernel:- Linear Kernel- Tree Kernel
Tree Kernel- very slow
![Page 16: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/16.jpg)
Experiments – Linear KernelTraining data are cut into slices.
Slice i contains two pairwise samples of each sentence.
22 SVMs on 22 slices of training data.
2 days to train an SVM in a Pentium III 1.13Ghz.
![Page 17: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/17.jpg)
Results
![Page 18: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/18.jpg)
Conclusions
Using an SVM approach :
- achieving state-of-the-art
results
- SVM with linear kernel is
superior to tree kernel in speed
and accuracy.
![Page 19: An SVM Based Voting Algorithm with Application to Parse Reranking](https://reader035.fdocuments.us/reader035/viewer/2022081520/568161a3550346895dd15c87/html5/thumbnails/19.jpg)
T
n
o u
k
h a
Y
!