Intelligent Database Systems Lab N.Y.U.S.T. I. M. Chinese Word Segmentation and Statistical Machine...

Intelligent Database Systems Lab

N.Y.U.S.T.I. M.

Chinese Word Segmentation and Statistical Machine Translation

Presenter : Wu, Jia-Hao

Authors : RUIQIANG ZHANG , KEIJI YASUDA , EIICHIRO SUMITA

TOSLP (2008)

國立雲林科技大學National Yunlin University of Science and Technology


N.Y.U.S.T.I. M.

2

Outline

Motivation

Objective

Methodology Dictionary-based

CRF-based

Experiments

Conclusion

Personal Comments


N.Y.U.S.T.I. M.

Motivation

Chinese word segmentation is a necessary step in Chinese-English statistical machine translation.

However, there are many choices involved in creating a CWS system such as various specifications and CWS methods.

Ex 我們要發展中國家用電器我們　要　發展　中國　家用電器

我們　要　發展中國家　用　電器We Want to develop China’s Home electrical appliances.

We Want Developing country To use Electrical appliances.


N.Y.U.S.T.I. M.

Motivation

Chinese word segmentation is a necessary step in Chinese-English statistical machine translation.

However, there are many choices involved in creating a CWS system such as various specifications and CWS methods.

Chinese word segmentation Statistical machine translation

The ChineseName is called by Rome phonetic transcription


N.Y.U.S.T.I. M.

Objective

They created 16 CWS schemes under different setting to examine the relationship between CWS and SMT.

The authors also tested two CWS methods that dictionary-based and CRF-based approaches.

The authors propose two approaches for combining advantages of different specifications . A simple concatenation of training data.

Implementing linear interpolation of multiple translation models.


N.Y.U.S.T.I. M.

Methodology-Dictionary-based

The pure dictionary-based CWS does not recognize OOV words.

The authors combined N-gram language model with Dictionary-based word segmentation.

For a give Chinese character sequence , C=c0c1c2…cN

The word sequence , W=wt0wt1wt2…wtM

Which satisfies

Out-of-vocabulary

10100...,... 10 ttttt ccwccw

MMMiii tttttt ccwccw ...,... 11 11

MiNttt iii 0,0,1 ),...()...,...(

),...()...(maxarg

)|()(maxarg)|(maxarg

11

0

1110

0010

MM

M

tMtttt

tttttW

WW

wccwcc

wccwwwP

WCPWPCWPW

δ(u,v) equal to 1 if both arguments are the same , and 0 otherwise.


N.Y.U.S.T.I. M.

Methodology-CRF-based IOB Tagging

Each character of a word is labeled. B if it is the first character of a multiple-character word.

O if the character functions as an independent word

I for other.

Ex ：全北京市 is labeled 全 /O 北 /B 京 /I 市 /I

The probability of an IOB tag sequence, T=t0t1…tM , given the word sequence W=w0w1…wM

Unigram features : w0,w-1,w1,w-2,w2,w0w-1,w0w1,w-1w1,w-2w-1,w2w0bigram features : simply used absolute counts for each feature in the training data and define a cutoff value for each feature type.


N.Y.U.S.T.I. M.

Methodology-Achilles

An In-House CWS including Both Dictionary-Based and CRF-Based Approaches. Dictionary-based

Zero OOV recognition rate. In-vocabulary rate is higher.

CRF-based

OOV recognition rate higher than Dictionary-based. Best F-scores.


N.Y.U.S.T.I. M.

Methodology-Phrase-Based SMT

The method use a framework of log-linear models to integrate multiple features.

Where fi(F,E) is the logarithmic value of the i-th feature ,and λi is the weight of the i-th feature. The target sentence candidate that maximizes P(E|F) is the solution.


N.Y.U.S.T.I. M.

Experiments

The data used in the experiments were provided by LDC , and use the English sentences of the data plus Xinhua news of the LDC Gigaword English corpus.

Implementation of CWS Schemes Tokens : the total number of words in the training data

Unique word : lexicon size of the segmented training data.

OOVs : the unknown words in the test data.


N.Y.U.S.T.I. M.

Experiment

The effect of CWS specifications on SMT.


N.Y.U.S.T.I. M.

Experiment


N.Y.U.S.T.I. M.

Experiment - Combining multiple CWS schemes Effect of Combining Training Data from Multiple CWS

Specifications. Create a new CWS scheme called dict-hybrid by combining AS,

CITYU, MSR, PKU.

49,546,231 tokens , 112,072 unique words for the training data. 693 OOVs for the test data.


N.Y.U.S.T.I. M.

Experiment

Effect of Feature Interpolation of Translation Models. The authors generated multiple translation models by using different

word segmenters.

The phrase translation model p(e|f) can be linearly interpolated as

Where pi(e|f) is the phrase translation model corresponding to the i-th CWSs. αi is the weight and S is the total number of models.


N.Y.U.S.T.I. M.

Conclusion

The authors analyzed multiple CWS specifications and built a CWS for each one to examine how they affected translations.

They proposed a new approach to linear interpolation of translation features , and improvement in translation and achieved the best BLEU score of all the CWS schemes.


N.Y.U.S.T.I. M.

Comments

Advantage There are many experiments to evaluate their performance.

Drawback But some interpretation of experiments are complex.

Application Chinese Word Segmentation.

Statistical Machine Translation.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. Chinese Word Segmentation and Statistical Machine...

Documents

Transcript of Intelligent Database Systems Lab N.Y.U.S.T. I. M. Chinese Word Segmentation and Statistical Machine...