Lucia Specia - SMT e pós-edição

Statistical Machine Translation andPost-editing

I Conferencia Internacional de Traducao e TecnologiaPorto, 14 Maio 2013

Lucia Specia

(Thanks to Wilker Aziz for many of the SMT slides)Department of Computer Science

University of Sheffieldhttp://www.dcs.shef.ac.uk/~lucia/

Porto (14/05/2013) SMT and PE 1 / 67

http://www.dcs.shef.ac.uk/~lucia/

Overview

• Machine Translation - task and challenges

• Statistical Machine Translation� Word-based models� Phrase-based models� Syntax-based models

• Post-editing:� Process and research� Practice with PET

Porto (14/05/2013) SMT and PE 2 / 67

The task of Machine Translation (MT)

The boy ate an apple

O menino comeu uma maca

BUT

He said that the bottle floated into the cave

? Dijo que la botella entro a la cueva flotando

虽然北风呼啸 , 但天空依然十分清澈。

However , the sky remained clear under the strong north wind .

Porto (14/05/2013) SMT and PE 3 / 67

Challenges in MT: Ambiguity

• Lexical ambiguity: different words meanings may translate differently

e.g. book the flight ⇒ reservar

read the book ⇒ libro

e.g. kill a man ⇒ matar

kill a process ⇒ acabar

• Syntactic ambiguity:

e.g. John hit the dog with the stick

⇒ John golpeo el perro con el palo

⇒ John golpeo el perro que tenia el palo

• Pronoun resolution:e.g. The computer outputs the data; it is fast

⇒ La computadora imprime los datos; es rapida VS.

e.g. The computer outputs the data; it is stored in ascii

⇒ La computadora imprime los datos; estan almacendos en ascii

Porto (14/05/2013) SMT and PE 4 / 67

Challenges in MT: Divergences

• Structural divergences:

e.g. The bottle floated into the cave

⇒ La botella entro a la cueva flotando(The bottle entered the cave floating)

• Different word orders:

� English is: subject – verb – objectJapanese is: subject – object – verb

English: IBM bought Lotus yesterdayJapanese: IBM Lotus bought yesterday

Porto (14/05/2013) SMT and PE 5 / 67

Challenges: Non-literal meaning

• Idioms

e.g. He finally kicked the bucket at the hospital

⇒ Ele finalmente chutou o balde no hospital VS.⇒ Ele finalmente bateu as botas no hospital

• Multi-word expressions

e.g. Do take the long waiting list for organ donation in this country intoaccount

⇒ Considere a longa lista de espera para doacao de orgaos neste paıs

Porto (14/05/2013) SMT and PE 6 / 67

Requires deep language processing

• One of the most challenging problems in NLP

• Requires� Full understanding of source text (analysis)� Generation of a coherent, accurate, fluent target text (synthesis)

Porto (14/05/2013) SMT and PE 7 / 67

Statistical Machine Translation

P(French|English)

• Statistical Machine Translation (SMT) learn how to generatetranslations using statistical methods

• Early 1990s by IBM. Adopted by Google, etc.

• Idea is older:

Warren Weaver (1949)

When I look at an article in Russian, I say: “This is really written inEnglish, but it has been coded in some strange symbols. I will nowproceed to decode.”

� Inspired by WWII code-breaking� Approach not feasible with early computers – idea not pursued

Porto (14/05/2013) SMT and PE 8 / 67

Probabilistic Model

• Output of decoder depends probabilistically on the input

• To translate French (F ) into English (E ):

Given a French sentence F search for English sentence E ∗ that maximisesP(E |F )

Porto (14/05/2013) SMT and PE 9 / 67

Probabilistic Model

• Find English sentence that maximises P(E |F ), i.e.

E ∗ = argmaxE

P(E |F )

= argmaxE

P(E)·P(F |E)P(F ) Bayes Rule

� P(F ) constant across different E , so:

E ∗ = argmaxE

P(E ) · P(F |E )drop constantdivisor P(F)

Porto (14/05/2013) SMT and PE 10 / 67

Noisy Channel Model & SMT (ctd)

E ∗ = argmaxE

P(E |F ) = argmaxE

P(E ) · P(F |E )

• Why not model P(E |F ) directly?

� Would need very good probability estimates

� Divide and conquer!

• Decomposition to P(E ) · P(F |E ) allows one to use less reliableprobabilities

� P(E ) worries about good English — fluency

• ensures that E has words in the right order and fit together

� P(F |E ) worries about French that translates English — faithfulness

• ensures that E has words that generally translate F

� P(E ) and P(F |E ) can be trained independently

Porto (14/05/2013) SMT and PE 11 / 67

Main Components for Translation (F 7→ E )

• Translation model (TM): P(F |E )

� Faithfulness: TMs learned by analysing large amounts of parallel text

• Judge which (F ,E) pairs are likely to be translations of each other

• Language model (LM): P(E )

� Fluency: LMs created from large volume of (fluent) target languagesentences

• Learn which word sequences are common/more probable in E

• A Decoder: (argmax)

� Search algorithm to find E∗

• V. large search space — heuristic search approach used

Porto (14/05/2013) SMT and PE 12 / 67

Learning Translation Models - P(F |E )

• Requires a bilingual corpus

� e.g. European/Canadian/Hong Kong parliaments, subtitles, Bible

• Bilingual corpus must be sentence-aligned:

• Can we estimate P(F |E ) from whole sentences?

Porto (14/05/2013) SMT and PE 13 / 67

Learning Translation Models - Word-based SMT

• Break sentences into smaller units: words are a good starting point

• Learn translation probabilities by word aligning a sentence-alignedcorpus:

Zenish

Uh usehUh jejjeYiguo useh

English

A homeA gardenI arrived home

Porto (14/05/2013) SMT and PE 14 / 67




Zenish


• The same word happens in source 1 and 3

English


Porto (14/05/2013) SMT and PE 15 / 67




Zenish


• Could we expect the same in the target side?

English


Porto (14/05/2013) SMT and PE 16 / 67




Zenish


English


• useh = home

Porto (14/05/2013) SMT and PE 17 / 67




Zenish


• What about the contexts?

English


Porto (14/05/2013) SMT and PE 18 / 67




Zenish


English


• We can align them: Yiguo = I; Yiguo = arrived; Uh = A

Porto (14/05/2013) SMT and PE 19 / 67




Zenish


English


• And reuse this knowledge to align more sentences: Uh = A

Porto (14/05/2013) SMT and PE 20 / 67




Zenish


English


• And again: jejje = garden

Porto (14/05/2013) SMT and PE 21 / 67

Learning Translation Models - Word-based SMT (ctd)

Word-alignment:

• Identify correspondences between two languages at the word level

• Basis for word-based SMT, first step for other approaches

• Alignment can be learned using Expectation Maximization (EM)� Treat all alternative word alignments as equally likely

� Observe across sentences that Zenish useh often links to English home• Increase probability of this word pair aligning• Has knock-on effect for alignment of other words

� Iteratively redistribute probabilities, until they identify most probablelinks for each word (convergence)

Word alignment commonly done using the IBM Models 1-5

Porto (14/05/2013) SMT and PE 22 / 67


• IBM models produce a probabilistic dictionary for the whole parallelcorpus:

contras contras 0.0285714contras los 0.0000011contras esos 0.0000812richer enriquecida 0.0476190richer mucho 0.0000502richer enriqueci 0.0769231richer variado 0.0066225

Porto (14/05/2013) SMT and PE 23 / 67


Back to the translation model:

At translation (decoding time), for a new sentence to translate, take theset of translations that jointly maximise the whole translation probability

E ∗ = argmaxE

P(F |E )

• Generates alternative translations using different translation optionsfor words and different word orders

• Selects the one with maximum score: no notion of context. Howabout ambiguity?

What about the context and the fluency in the target language?

Porto (14/05/2013) SMT and PE 24 / 67

Learning Language Models - Word-based SMT

Language model: P(E )

E ∗ = argmaxE

P(F |E ) · P(E )

• Takes different translation options, tests how “common” each ofthem are in the target language as a whole sentence (some context)

• P(E ) = probability of strings E based on relative frequencies in alarge corpus of language E

Google

the house is small 144,000the house is little 85,300small house 2,330,000little house 3,330,000

Porto (14/05/2013) SMT and PE 25 / 67

Learning Language Models - Word-based SMT (ctd)

• Model P(E ) by decomposing into single-word probabilities givenprevious words, under Markov assumption: only the previous n-1words matter for predicting a word. E.g for trigram models, n = 3

' P(e1) · P(e2|e1) · P(e3|e1, e2) · P(e4|e2, e3) · · ·P(en|en−2, en−1)

• Relative frequencies to compute these probabilities. E.g. trigrams:

P(e3|e1, e2) = count(e1e2e3)count(e1e2)

P(little|house, is) = count(house is little)count(house is)

Porto (14/05/2013) SMT and PE 26 / 67

Word-based SMT – Limitations

• It is difficult to do alignment, and hence learn a TM, for languagesthat are structurally quite different, with different words orders� Considering all possible word orders – too costly� Poor reordering model: flexibility vs noise

• Some languages may have different notions of what counts as a word

Donaydampfshiffahrtsgesellschaftskapitaenskajuetenschluesseloch

The keyhole of the door of the cabin of the captain of a steamshipcompany operating on the Danubea

ahttp://www.chebucto.ns.ca/~waterbuf/longwrds.html

Porto (14/05/2013) SMT and PE 27 / 67

http://www.chebucto.ns.ca/~waterbuf/longwrds.html

Phrase-based SMT

• Leading approach since 2003

No voy1 a la2 casa3 → I am not going1 to the2 house3

it seems to1 me2 → me1 parece2

Je1 ne vais pas2 a la3 maison4 → I1 am not going2 to the3 house4

Eu1 sinto saudade de voce2 → I1 miss you2

I1 miss you2 → Eu1 sinto sua falta2

natuerlich1 hat2 John3 spass am4 spiel5 → of course1 John2 has3 fun with the4 game5

• More intuitive and reliable alignments� Account for reordering within the phrases� Phrases can still be reordered

• Can we learn phrase translation probability distribution from theparallel corpus using EM?

� 1 src-tgt sentence → O(n4) phrase pairs

Porto (14/05/2013) SMT and PE 28 / 67

Phrase-based SMT - Phrases from word alignments

• Extract phrase pairs that are consistent with the word alignment• Phrase: sequence of tokens, not linguistically motivated• Word alignment produced by any of the IBM Models - only done once

reanudacion del perıodo de sesiones

resumption

of

the

session

1 resumption ↔ reanudacion

2 of the ↔ del

3 session ↔ perıodo de sesiones

4 resumption of the ↔ reanudacion del

5 of the session ↔ del sesiones

6 resumption of the session ↔ reanudacion del sesiones

Porto (14/05/2013) SMT and PE 29 / 67

Phrase-based SMT - Computing phrase probabilitydistributions

1 Extract phrase pairs from word aligned parallel corpus√

2 Extract counts of those phrases from large parallel corpus:

φ(f |e) =count(f , e)

count(e)

3 Store phrases and their probabilities in a phrase table� Probabilistic dictionary of phrases

Porto (14/05/2013) SMT and PE 30 / 67

Phrase-based SMT - Limitation of Noisy Channel Model

E ∗ = argmaxE

P(F |E ) · P(E )

• Which component is more important?

� P(F |E ) ?� P(E ) ?

• Depends on the corpus used (more parallel data makes P(F |E ) morereliable), language-pair, etc.

Porto (14/05/2013) SMT and PE 31 / 67

Phrase-based SMT - Log-linear Model (ctd)

Assign weights λi to components (feature functions) hi (f, e):

e∗ = argmaxe

∑ni=1 λihi (f, e)

Components

1 P(E )

2 P(F |E )

Weights

1 λP(E)

2 λP(F |E)

Feature Functions

1 h1 = logP(E)

2 h2 = logP(F |E)

Benefits

1 Extensible: other components are used: e.g. reordering, word andphrase penalties

2 Weights can be tuned, i.e., learned from examples to minimise errorin a subset of the parallel corpus, e.g. MERT

Porto (14/05/2013) SMT and PE 32 / 67

Phrase-based SMT - Decoding (ctd)

Search problem: segmenting source sentence and selecting phrasetranslations:

• All phrases matching source words are selected from the phrase table

J’ ai les yeux noirs .I have the eyes black .

me has them eye dark ,

I have eyes espresso !

I am the eyes somber .

I did some black eyes .

I had black eyes .

I have black eyes .

black eyes I have .

• The goal of the decoder is to select the phrases whose combination(in a given order) yields the highest score for the log-linear model

Porto (14/05/2013) SMT and PE 33 / 67


Searching = incrementally construct the translation hypotheses by tryingout several possibilities

• Generating target words in sequence, from left to right

• Incrementally compute the overall log-linear model score for eachhypothesis

• Heuristics to prune search space, e.g. stack-based beam search� Use one stack for of every length of partial hypotheses� Limited space for partial hypotheses, only keep those that are

promising, e.g. model score is close to that of the best partialhypothesis so far

Porto (14/05/2013) SMT and PE 34 / 67


Search space

Hypothesis

• Covered source words

• Target (output) words

• Model score (using the combined probabilities)

Porto (14/05/2013) SMT and PE 35 / 67

Hierarchical and Syntax-based SMT

• PBSMT still has trouble with long-distance reorderings

• Ways of bringing structure and linguistic knowledge into the transferrules of the phrase table

Porto (14/05/2013) SMT and PE 36 / 67

Hierarchical SMT - Motivation

Introduce structure into phrase-based SMT models to deal withlong-distance reordering

Ich

wer

de

Ihn

en

die

ents

pre

chen

den

An

mer

ku

ng

en

au

sha

nd

igen

Ishallbepassingontoyousomecomments

Porto (14/05/2013) SMT and PE 37 / 67



Ich

wer

de

Ihn

en

die

ents

pre

chen

den

An

mer

ku

ng

en

au

sha

nd

igen


• How can we get a phrase for shallbe passing on?

Porto (14/05/2013) SMT and PE 38 / 67



Ich

wer

de

Ihn

en

die

ents

pre

chen

den

An

mer

ku

ng

en

au

sha

nd

igen

Ishallbepassingonto Xyou Xsome Xcomments X


• We cannot, unless we get to yousome comments along

Porto (14/05/2013) SMT and PE 39 / 67



Ich

wer

de

Ihn

en

die

ents

pre

chen

den

An

mer

ku

ng

en

au

sha

nd

igen



• We cannot, unless we get to yousome comments along

• Unless we replace all those wordsby a variable

Porto (14/05/2013) SMT and PE 40 / 67

Hierarchical SMT - Motivation (ctd)

shall be passing on to you some commentsl

werde Ihnen die entsprechenden Anmerkungen aushandigen



shall be passing on Xl

werde X aushandigen

Porto (14/05/2013) SMT and PE 41 / 67

Hierarchical SMT - basics

Learnt from word-aligned parallel corpora in the same way as before

• Based on the fact that language has recursive structures

• Phrases within other phrases treated as nonterminals: replaced by X

• No linguistic constraints added - yet, some structure

Porto (14/05/2013) SMT and PE 42 / 67

Hierarchical SMT - basics (ctd)



shall be passing on X1 some commentsl

werde X1 die entsprechenden Anmerkungen aushandigen

shall be passing on X1 X2

lwerde X1 X2 aushandigen

Porto (14/05/2013) SMT and PE 43 / 67

Hierarchical SMT - phrase-table

[X ]→ shall be passing on X1 X2 | werde X1 X2 aushandigen

[X ]→ shall be passing on X3 | werde X3 aushandigen

[X ]→ to you | Ihnen

[X ]→ some comments | die entsprechenden Anmerkungen

[X ]→ to you some comments | Ihnen die entsprechenden Anmerkungen

Learn a bilingual (synchronous) set of context-free rules about how totranslate F into E

Porto (14/05/2013) SMT and PE 44 / 67

Syntax-based SMT

• Hierarchical models are not very informative (Xs) and suffer from anexponential number of rules

• Syntax-based SMT overcomes these limitations by using linguisticcategorise to label nodes

• A syntactic parser is required at pre-processing time on at least onelanguage (the other could have Xs)

• Idea:� Learn a bilingual (synchronous) set of linguistically informed

context-free rules about how to translate F into E� Rules are learnt from the word-alignment, constrained by syntactic

categories

Porto (14/05/2013) SMT and PE 45 / 67

Syntax-based SMT (ctd)

S

PRP

J’

VP

VB

ai

NP

DT

les

NN

yeux

JJ

noirs

S

PRP

I

VP

VB

have

NP

JJ

black

NN

eyes

J’ ai les yeux noirs

I

have

black

eyes

Standard constraints on rule construction:

• Single nonterminal on the left

• Consistent with word-alignment

• Nonterminals on the right must alignone-to-one

Porto (14/05/2013) SMT and PE 46 / 67

Summary on SMT

• Adding linguistic knowledge to SMT is a trend:� Stronger theoretical motivation� Deals with reordering, sparsity, ambiguity� Depends on (quality of) parsers� Higher complexity

• Performance:� Hard to quantify in absolute terms� Most language pairs → phrase-based or hierarchical SMT does better� Language pairs with long-distance reordering → syntax-based SMT

does better

• Other open issues:� Limited use of linguistic information - at most syntax, some attempts

to use shallow semantics - not yet there� Limited use of context - sentence-level� Domain discrepancies - mistranslations, unknown words

• Overall, to reach publishable quality, human checking/revisionnecessary: post-editing

Porto (14/05/2013) SMT and PE 48 / 67

Definitions

Note: Some slides are based on those by Sharon O’Brien (tutorial on MTpost-editing at AMTA-2010).

What is post-editing?

• Same as translation?

• Same as revision?

• A new profession? New skills?

Who does the post-editing?

What are the quality expectations?

Porto (14/05/2013) SMT and PE 49 / 67

Degrees/types of post-editing

• Light (fast/rapid/gist) post-editing: essential corrections

• Full post-editing: more corrections, higher quality, closer to revision

• Monolingual post-editing: no access to source text

Decided by:

• User requirements

• Volume

• Quality expectations

• Turn-around time, etc.

Porto (14/05/2013) SMT and PE 50 / 67

Quality of post-editing

• Can MT+PE (possibly +revision) produce the same level of qualityas HT+revision?

• Dependent on the quality of MT systems: variable for language pair,text type/genre & domain, quality of source

• Dependent on quality requirements

• Different outcomes:� Autodesk reports lower error rates with MT+PE [Aut11]� Tilde observed higher error rate with MT+PE [SPSV11] -

under-resourced highly-inflected languages: lower MT quality

Porto (14/05/2013) SMT and PE 51 / 67

Ways of measuring quality of MT for PE

• Automatic MT evaluation metrics ? BLEU/TER/METEOR... -automatic, but do not correlate well with PE effort

• Error counts: compare source with raw MT output - manual, costly� Can we use the “standard” error typologies for HT? Do MT and HT

make same mistakes?

QTLaunchPad project

Multidimensional Quality Metrics for MT and HT, for manual and(semi-)automatic evaluation (QE): http://www.qt21.eu/launchpad/

• % Changes made: compare post-edited text with raw MT output -automatic

• Time to make changes: record time to fix raw MT - automatic

• Estimated effort: take source text and raw MT output and estimatePE effort prior to post-editing (QE) - automatic, less precise� Is this the same as fuzzy match level in TM systems?

Porto (14/05/2013) SMT and PE 52 / 67

http://www.qt21.eu/launchpad/

Aspects involved in post-editing effort

• Temporal: time to post-edit (variable across translators)

• Technical: no./% edits made, keys pressed

• Cognitive: time, pauses, movements

Studies on PE process:

• Is PEMT faster than HT? Yes

• Is PEMT less keyboard intensive than HT? Yes

• Is PEMT less cognitively demanding than HT?

� Cognitive effort is the most difficult to estimate: active research area[Kop12, KARS12]

• PE more tedious/tiring? Mixed opinions

Check work by

Michael Carl - CBS (Translog) and Sharon O’Brien - DCU (eye-tracking)

Porto (14/05/2013) SMT and PE 53 / 67

Pricing models

Open question. Some common practices:

• Paying as fuzzy segment matches (source-based)

• Paying as % of edits performed (MT&PE-based)

• Paying a fee based on time spent (less common)

Porto (14/05/2013) SMT and PE 54 / 67

PE Guidelines

• No standard guidelines, but good practices (e.g. TAUS/CNGL)

• Guidelines need to clearly specify level of post-editing and qualityexpectations

• Guidelines need to clearly differentiate between essential andpreferential changes

• Ideally, guidelines should be system-, language-, purpose-specific

Porto (14/05/2013) SMT and PE 55 / 67

General PE Guidelines for “good enough” quality -TAUS/CNGL

• Aim for semantically correct translation.

• Ensure that no information has been accidentally added or omitted.

• Edit any offensive, inappropriate or culturally unacceptable content.

• Use as much of the raw MT output as possible.

• Basic rules regarding spelling apply.

• No need to implement corrections that are of a stylistic nature only.

• No need to restructure sentences solely to improve the natural flow ofthe text.

http://www.translationautomation.com/postediting/

machine-translation-post-editing-guidelines

Porto (14/05/2013) SMT and PE 56 / 67

http://www.translationautomation.com/postediting/machine-translation-post-editing-guidelines


General PE Guidelines for HT quality - TAUS/CNGL

• Aim for grammatically, syntactically and semantically correcttranslation.

• Ensure that key terminology is correctly translated and thatuntranslated terms belong to the clients list of Do Not Translateterms.

• Ensure that no information has been accidentally added or omitted.

• Edit any offensive, inappropriate or culturally unacceptable content.

• Use as much of the raw MT output as possible.

• Basic rules regarding spelling, punctuation and hyphenation apply.

• Ensure that formatting is correct.

http://www.translationautomation.com/postediting/

machine-translation-post-editing-guidelines

Porto (14/05/2013) SMT and PE 57 / 67



IT PE Guidelines - Midori Tatsumi’s PhD thesis

• What needs to be fixed:� Non-translatable items, such as command and variable names, that

have been translated. Please put it back to English.� Inappropriately translated general IT terms.� Mistranslation.� Word orders that are inappropriate to the level that the sentence has

become impossible or difficult to comprehend.� Comprehensible but extremely unnatural or inappropriate expressions.� Inappropriate postpositions and conjugations.

• What are preferred to be standardised:� Repetitive items, such as procedures and steps.� Styles of the section titles in a user manual and help files.

• What does not need to be fixed:� Ending styles of bulleted items.� Placeholders, such as &ProductNameShort; and &ProductNameLong;� Punctuation.

http://doras.dcu.ie/16062/1/SAKURA_final_revised.pdfPorto (14/05/2013) SMT and PE 58 / 67

http://doras.dcu.ie/16062/1/SAKURA_final_revised.pdf

Monolingual PE Guidelines - ACCEPT project

• Try and edit the text by making it more fluent and clearer based onhow you interpret its meaning. For example, try to rectify word orderand spelling when they are inappropriate to the extent that the texthas become impossible or difficult to comprehend

• If words, phrases, or punctuation in the text are completelyacceptable, try and use them (unmodified) rather than substitutingthem with something new and different.

http://www.accept.unige.ch/Products/D6.2.2_Seminar_

Material_on_Post-Editing_Edition_2.pdf

Porto (14/05/2013) SMT and PE 59 / 67

http://www.accept.unige.ch/Products/D6.2.2_Seminar_Material_on_Post-Editing_Edition_2.pdf

http://www.accept.unige.ch/Products/D6.2.2_Seminar_Material_on_Post-Editing_Edition_2.pdf

PE Technology (tools)

A myriad of tools appeared in the marked in the last couple of years:

• TM tools that now integrate MT, e.g. SDL TRADOS, WordFast

• MT tools (with or without TM): e.g. Systran, Google TranslateToolkit, Caitra

• PE tools that can use different MT systems: MemoQ, MemSource,SmartMate, Coach, ...

More and more functionalities: spell/grammar checkers, multiple MT,MT+TM, interactive MT, terminology, word/phrase alignment, etc.Limited (if any) logging of translator’s activities for productivity analysis –not enough for research.

Porto (14/05/2013) SMT and PE 60 / 67

PE Technology (tools)

Research-oriented tools:

• Translog (not open source)

• OmegaT (TM only)

• Appraise (MT eval through ranking, no detailed statistics for PE)

• PET: http://pers-www.wlv.ac.uk/~in1676/pet/index.html

• Coming soon: open source tool promised as part of EU projectsCASMACAT & MATECAT

Porto (14/05/2013) SMT and PE 61 / 67

http://pers-www.wlv.ac.uk/~in1676/pet/index.html

What can we get from PET?

Comparison between HT and multiple MT systems [dSAS11]

• PET to compare 3 MT systems vs translation from scratch

• Sitcom and movie subtitles:� Translating from scratch can be 73% slower than post-editing� SMT systems (Google and Moses) performed the best

How often a system produced an output that was faster to PE than other systems:System Google Moses Systran Trados

Google - 139 161 187

Moses 69 - 122 164

Systran 69 106 - 145

Trados 48 67 89 -

How often post-editing a system output was faster than translating:System Faster than human translation

Google 94%Moses 86.8%

Systran 81.20%Trados 72.40%

Porto (14/05/2013) SMT and PE 62 / 67

What can we get from PET?

Evaluation of an MT system for translation of subtitles (space constraints)[AdSS12]

• Add compression constraints to an MT system to generate lengthcompliant subtitles

• PET to guide post-editing according to length and time requirementsfor every unit� Show space limitation (attribute)� Change colour of translation if too long� Offer shorter paraphrases

• PET to evaluate edit distance and length

SystemDexter How I Met.. Terra Nova

TER ↓ LENGTH TER ↓ LENGTH TER ↓ LENGTHMosest 30.3 116.0 20.0 108.5 33.8 120.2Google 63.6 156.5 52.8 144.3 63.1 152.1MosesLP2 29.5 115.5 21.0 109.1 33.4 119.3MosesLP1 28.3 115.8 20.7 110.0 34.8 119.8

Porto (14/05/2013) SMT and PE 63 / 67

Hands-on activity with PET

• en-pt-hiding-and-assessing.pec – bigbang-pe-2.pej

• en-pt-hiding-and-assessing.pec – bigbang-ht-2.pej

• fapesp.pec – fapesp-annotated.pej

• fapesp-mono.pec – fapesp-annotated.pej

• subs.pec – greys.08x06.v1-v1.en-br.s2.1.pej

Advanced use:

• Configuration file: .pec

• Input/output XML files

• How to build a PEJ file from raw files: pej.bat

• How to interpret the results: scripts/readper.pl

Porto (14/05/2013) SMT and PE 64 / 67

For more on the research-side

Workshop on Post-Editing Technology and Practice:

• AMTA 2012: https://sites.google.com/site/wptp2012/

• MT Summit 2013 (deadline for papers 28/05/2013):https://sites.google.com/site/mts2013wptp/

Porto (14/05/2013) SMT and PE 65 / 67

https://sites.google.com/site/wptp2012/

https://sites.google.com/site/mts2013wptp/

References I

Wilker Aziz, Sheila Castilho Monteiro de Sousa, and Lucia Specia.

Cross-lingual sentence compression for subtitles.

In The 16th Annual Conference of the European Association for MachineTranslation, EAMT ’12, pages 103–110, Trento, Italy, May 2012.

Autodesk.

Translation and Post-Editing Productivity.

In http: // translate. autodesk. com/ productivity. html , 2011.

Sheila C. M. de Sousa, Wilker Aziz, and Lucia Specia.

Assessing the post-editing effort for automatic and semi-automatictranslations of DVD subtitles.

In Conference Recent Advances in Natural Language Processing 2011, pages97–103, Hissar, Bulgaria, 2011.

Porto (14/05/2013) SMT and PE 66 / 67

http://translate.autodesk.com/productivity.html

References II

Maarit Koponen, Wilker Aziz, Luciana Ramos, and Lucia Specia.

Post-editing time as a measure of cognitive effort .

In AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP2012), pages 11–20, San Diego, USA, October 2012. Association forMachine Translation in the Americas (AMTA).

Maarit Koponen.

Comparing human perceptions of post-editing effort with post-editingoperations.

In Proceedings of the Seventh Workshop on Statistical Machine Translation,pages 181–190, Montreal, Canada, 2012.

R. Skadins, M. Purins, I. Skadina, and A. Vasiljevs.

An evaluation of SMT in localization to under-resourced inflected languages.

In Proceedings of the 15th Conference of the European Association forMachine Translation, Leuven, 2011.

Porto (14/05/2013) SMT and PE 67 / 67

Lucia Specia - SMT e pós-edição

Technology

Transcript of Lucia Specia - SMT e pós-edição