Language & K nowledge Engineering Lab

A Probabilistic Framework for Structure-based

AlignmentKurohashi-lab M2

56430 Toshiaki Nakazawa

Language & Knowledge Engineering Lab

OutlineI. Introduction of Machine Translation

i. What is Alignment?ii. Statistical Machine Translation (SMT)iii. Example-based Machine Translation (EBMT)

II. Baseline alignment methodIII. A probabilistic framework for alignment

i. Corresponding Pattern score (CP-score)ii. Integration of Maximum Entropy (ME)

IV. Experiments and resultsV. Discussion and conclusion

Language & Knowledge Engineering Lab

OutlineI. Introduction of Machine Translation

i. What is Alignment?ii. Statistical Machine Translation (SMT)iii. Example-based Machine Translation (EBMT)

II. Baseline alignment methodIII. A probabilistic framework for alignment

i. Corresponding Pattern score (CP-score)ii. Integration of Maximum Entropy (ME)

IV. Experiments and resultsV. Discussion and conclusion

Language & Knowledge Engineering LabStandard Way of Machine Translation

ParallelCorpus Alignment Resource

Output

Translation

Input

Parallel Corpus: Text which is written in two different languages but the content is almost same.Alignment: To find the correspondence between two parallel sentences. (word level, phrase level, etc…)

The performance of alignment affects the accuracy of

translation.

Language & Knowledge Engineering LabStatistical Machine Translation (SMT)

Learn models for translation from parallel corpus statistically

Not use any linguistic resources Small translation unit (= “word”)

– Recently, the number of studies handling bigger unit (= “couple of words” or “phrase”) is increasing

Require large parallel corpus for highly-accurate translation

Language & Knowledge Engineering Lab

Basic Method for SMT Translate by maximizing the probability:

)|()(maxarg

)|(maxarg

EJPEP

JEPE

E

E

Language Model Translation Model

Learn from a parallel corpus(usually with unsupervised learning algorithm)

Ex) IBM Model [Brown et al., 93]

Language & Knowledge Engineering Lab

Overview of EBMT

ParallelCorpus Alignment TMDB

Output

Translation

Input

Advanced NLP technologies

Translation Memory Data

Base

Language & Knowledge Engineering LabExample-based Machine Translation (EBMT)

Divide the input sentence into a few parts Find a similar expressions (examples)

from parallel corpus for each parts Combine the examples to generate output

translation Use any linguistic resources as much as

possible Larger translation unit (larger example) is

better

Language & Knowledge Engineering Lab

Flow of EBMT

Language & Knowledge Engineering Lab

SMT vs. EBMT

SMT EBMT

GoodPoint

- Works enough for languages which don’t have sufficient NLP resources.

- Active to utilize any kinds of NLP resources.- High performance.

BadPoint

- Not easy to achieve high performance.- Weak for the wide difference between the languages.

- Algorithm is usually heuristic.- Modification is necessary for each language pair.

We introduce a probabilistic framework for structure-

based alignment.

Language & Knowledge Engineering Lab

OutlineI. Introduction of Machine Translation

i. What is Alignment?ii. Statistical Machine Translation (SMT)iii. Example-based Machine Translation (EBMT)

II. Baseline alignment methodIII. A probabilistic framework for alignment

i. Corresponding Pattern score (CP-score)ii. Integration of Maximum Entropy (ME)

IV. Experiments and resultsV. Discussion and conclusion

Language & Knowledge Engineering Lab

Alignment

交差点 で 、突然

あの車 が

飛び出して 来た のです

the carcame

at mefrom the side

at the intersection

1. Transformation into dependency structure

J: 交差点で、突然あの車が 飛び出して来たのです。

E ： The car came at me from the side at the intersection.

J: JUMAN/KNPE: Charniak’s nlparser → Dependency tree

Language & Knowledge Engineering Lab

Alignment

交差点 で 、突然

あの車 が

飛び出して 来た のです

the carcame

at mefrom the side

at the intersection

1. Transformation into dependency structure2. Detection of word(s) correspondences

• Bilingual dictionaries• Transliteration detection

ローズワイン → rosuwain ⇔ rose wine (similarity:0.78)新宿 → shinjuku ⇔ shinjuku (similarity:1.0)

Language & Knowledge Engineering Lab

Alignment

交差点 で 、突然

あの車 が

飛び出して 来た のです

the carcame

at mefrom the side

at the intersection

1. Transformation into dependency structure2. Detection of word(s) correspondences3. Disambiguation of correspondences

Language & Knowledge Engineering Lab

Disambiguation

日本 で保険

会社 に対して

保険請求の

申し立て が可能です よ

you

will haveto file

insurance

an claim

insurance

with the office

in Japan

Cunamb → Camb : 1/(Distance in J tree) + 1/(Distance in E tree)

1/2 + 1/1

Language & Knowledge Engineering Lab

Alignment

交差点 で 、突然

あの車 が

飛び出して 来た のです

the carcame

at mefrom the side

at the intersection

1. Transformation into dependency structure2. Detection of word(s) correspondences3. Disambiguation of correspondences4. Handling of remaining phrases

Language & Knowledge Engineering Lab

Alignment

交差点 で 、突然

あの車 が

飛び出して 来た のです

the carcame

at mefrom the side

at the intersection

1. Transformation into dependency structure2. Detection of word(s) correspondences3. Disambiguation of correspondences4. Handling of remaining phrases5. Registration to translation example database

Language & Knowledge Engineering Lab

OutlineI. Introduction of Machine Translation

i. What is Alignment?ii. Statistical Machine Translation (SMT)iii. Example-based Machine Translation (EBMT)

II. Baseline alignment methodIII. A probabilistic framework for alignment

i. Corresponding Pattern score (CP-score)ii. Integration of Maximum Entropy (ME)

IV. Experiments and resultsV. Discussion and conclusion

Language & Knowledge Engineering Lab

Corresponding Pattern (CP)

Language & Knowledge Engineering Lab

Corresponding Pattern (CP)

Language & Knowledge Engineering Lab

Corresponding Pattern (CP)

(1, 2, 1, 1)

(0, 1, 0, 1)(0, 2, 0, 1) (0, 2) (0, 1) (0, 1) (0, 1)

(1, 2) (1, 1)

Language & Knowledge Engineering Lab

CP-score Assign a score to each CP = CP-score Calculation of CP-score

– Count the frequency of each CP Using the aligned parallel corpus by the baseline align

ment method

– Divide the frequency by the total frequency of all CPs (CP-score is a probability of occurrence)

Alignment Score (AS) by CP-score

1

1 1,

M

i

M

ijjiscoreCPAS

Language & Knowledge Engineering Lab

Alignment Disambiguation by AS

Adopt the alignment with highest AS

Language & Knowledge Engineering Lab

OutlineI. Introduction of Machine Translation

i. What is Alignment?ii. Statistical Machine Translation (SMT)iii. Example-based Machine Translation (EBMT)

II. Baseline alignment methodIII. A probabilistic framework for alignment

i. Corresponding Pattern score (CP-score)ii. Integration of Maximum Entropy (ME)

IV. Experiments and resultsV. Discussion and conclusion

Language & Knowledge Engineering Lab

Maximum Entropy (ME) The principle of maximum entropy:

– a method for analyzing the available information in order to determine a unique epistemic probability distribution. (by WIKIPEDIA)

Language & Knowledge Engineering Lab

Maximum Entropy (ME) The principle of maximum entropy:

– a method for analyzing the available information in order to determine a unique epistemic probability distribution. (by WIKIPEDIA)

Alignment probability with ME [Och et al,. 02]

),,( TSAmhS: Source sentenceT: Target sentenceA: Alignment

ATSA

TSATSA

]),,(exp[

]),,(exp[),|Pr(

1

1M

m mm

M

m mm

h

h

m

:Feature function

: Model parameter

Language & Knowledge Engineering Lab

Feature Functions

1. Alignment Score (AS)2. Parse score (Jap. and Eng.)3. Depth pattern score (DP-score)4. Probability of lexicon (Jap. and Eng.)5. Coverage of the correspondences (Jap.

and Eng.)6. Average size of the correspondences

(Jap. and Eng.)

Language & Knowledge Engineering Lab

OutlineI. Introduction of Machine Translation

i. What is Alignment?ii. Statistical Machine Translation (SMT)iii. Example-based Machine Translation (EBMT)

II. Baseline alignment methodIII. A probabilistic framework for alignment

i. Corresponding Pattern score (CP-score)ii. Integration of Maximum Entropy (ME)

IV. Experiments and resultsV. Discussion and conclusion

Language & Knowledge Engineering Lab

Experiments Select 500 moderately long sentences

from BTEC corpus of IWSLT2005 training data set

Manually annotate phrase-to-phrase alignment

Conducted 5-fold cross validation– 400 for training and 100 for testing

Calculated the F-measure

RPPRF

2 P: Precision

R: Recall

Language & Knowledge Engineering Lab

Results

Method

All sentences w/ function words

All sentences w/o function words

Ambiguous sentences w/ function words

Baseline 63.86 65.14 60.43

+CP-score 64.21 65.54 61.60

+ME 64.58 66.03 63.00

GIZA++ 22.14 52.85 23.78

Language & Knowledge Engineering Lab

Discussion Not considering clause

– Correspondences in the same clause of source sentence are likely to be in the same clause of target sentence

Sentence complexity– Proposed method works effectively for

long and complex sentences Preciseness of dictionary

– Erroneous correspondence by the dictionary makes bad effects on alignment

Language & Knowledge Engineering Lab

Conclusion Proposed a probabilistic framework to

improve structure-based alignment Proposed a new criteria CP-score for

evaluating alignment Integrate the ME model into alignment

approach

Language & Knowledge Engineering Lab

Future Work Sophisticate the CP and CP-score

– Consider clauses Select the feature functions Test our method on other corpora

– Longer and more complex sentences