Context-free translation models - mt-archive.info · Finite-State Models •Very simple models get...

Context-Free Translation Models

Adam LopezUniversity of Edinburgh → Johns Hopkins University

Finite-State Models

•Very simple models get us pretty far!

Finite-State Models

•There’s no data like more data.

Finite-State Models

•Word-based models follow intuitions, but not all.

Finite-State Models

•Phrase-based models are similar, but more effective.

Finite-State Models

•All of these models are weighted regular languages.

Finite-State Models

•Need dynamic programming with approximations.

Finite-State Models

•Need dynamic programming with approximations.

•Is this the best we can do?

Overview

training data(parallel text) learner model

联合国安全理事会的

五个常任理事国都decoder

However , the sky remained clear under the strong north wind .

Two Problems

•Exact decoding requires exponential time.

•This is a consequence of arbitrary permutation.

•But in translation reordering is not arbitrary!

•Parameterization of reordering is weak.

•No generalization!

Garcia and associates .

Garcia y asociados .Carlos Garcia has three associates .

Carlos Garcia tiene tres asociados .his associates are not strong .

sus asociados no son fuertes .Garcia has a company also .

Garcia tambien tiene una empresa .its clients are angry .

sus clientes estan enfadados .the associates are also angry .

los asociados tambien estan enfadados .

la empresa tiene enemigos fuertes en Europa .

the company has strong enemies in Europe .the clients and the associates are enemies .

los clientes y los asociados son enemigos .the company has three groups .

la empresa tiene tres grupos .its groups are in Europe .

sus grupos estan en Europa .the modern groups sell strong pharmaceuticals .

los grupos modernos venden medicinas fuertes .the groups do not sell zanzanine .

los grupos no venden zanzanina .the small groups are not modern .

los grupos pequenos no son modernos .

Same pattern:NN JJ → JJ NN

Finite-state models do not capture this generalization.

Context-Free Grammar

S → NP VPNP → watashi waNP → hako woVP → NP V V → akemasu

watashi wa

NP Vwatashi wa

hako wo

NP Vwatashi wa

hako wo

NP Vwatashi wa

hako wo akemasu

NP Vwatashi wa

hako wo akemasu

NP Vwatashi wa

hako wo akemasu

watashi wa hako wo akemasu

Synchronous Context-Free Grammar

S → NP VPNP → INP → the boxVP → V NP V → open

S → NP1 VP2 / NP1 VP2

NP → watashi wa / I NP → hako wo / the boxVP → NP1 V2 / V1 NP2

V → akemasu / open

NP VP NP VP

watashi wa I

NP VP NP VP

watashi wa I

NP VP NP VP

NP V V NPwatashi wa I

NP VP NP VP

hako wo the box

NP VP NP VP

hako wo the box

NP VP NP VP

akemasu openhako wo the box

Synchronous Context-Free GrammarS S

NP VP NP VP

watashi wa hako wo akemasu I open the box

Translation as Parsing

Translation as ParsingS

NP Vwatashi wa

akemasuhako wo

NP Vwatashi wa

akemasuhako wo

open the box

NP Vwatashi wa

akemasuhako wo

open the box

watashi wa hako wo akemasu I open the box

Decoding

•In general, there are an exponential number of possible parse trees for a sentence.

Decoding

•In general, there are an exponential number of possible parse trees for a sentence.

•Dynamic programming to the rescue!

Parsing

ParsingNN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

VB → duckSBAR → PRP VB

VP → VBD SBAR

Parsing

I1 saw2 her3 duck4

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

VP → VBD SBAR

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

VP → VBD SBAR

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

VP → VBD SBAR

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

VP → VBD SBAR

Parsing

I1 saw2 her3 duck4

PRP0,1 ! (w1 = I) " (PRP # I)Xi,i+1 ! (wi+1 = w) " (X # w)NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

VP → VBD SBAR

Parsing

I1 saw2 her3 duck4

PRP0,1 ! (w1 = I) " (PRP # I)Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

VP → VBD SBAR

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

VP → VBD SBAR

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

VP → VBD SBAR VBD1,2

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

Xi,j ! Yi,k " Zk,j " (X # Y Z)

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

Xi,j ! Yi,k " Zk,j " (X # Y Z)NP2,4 ! PRP$2,3 "NN3,4 " (NP# PRP$ NN)

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

Xi,j ! Yi,k " Zk,j " (X # Y Z)NP2,4 ! PRP$2,3 "NN3,4 " (NP# PRP$ NN)

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

NP2,4 SBAR2,4

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

NP2,4 SBAR2,4

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

NP2,4 SBAR2,4

Parsing

I1 saw2 her3 duck4

Xi,i+1 ! (wi+1 = w) " (X # w)

PRP0,1

NN → duck

PRP → I

VBD → saw

PRP$ → her

NP → PRP$ NN

VP → VBD NP

S → PRP VP

PRP → her

PRP$2,3

PRP2,3

NP2,4 SBAR2,4

Parsing

I1 saw2 her3 duck4

PRP0,1

VBD1,2

PRP$2,3 NN3,4

PRP2,3 VB3,4

SBAR2,4

Parsing

I1 saw2 her3 duck4

PRP0,1

VBD1,2

PRP$2,3 NN3,4

PRP2,3 VB3,4

SBAR2,4

Parsing

I1 saw2 her3 duck4

PRP0,1

VBD1,2

PRP$2,3 NN3,4

PRP2,3 VB3,4

SBAR2,4

I saw her duck

PRP VBD PRP$ NN

Parsing

I1 saw2 her3 duck4

PRP0,1

VBD1,2

PRP$2,3 NN3,4

PRP2,3 VB3,4

SBAR2,4

I saw her duck

PRP VBD PRP VB

Parsing

I1 saw2 her3 duck4

PRP0,1

VBD1,2

PRP$2,3 NN3,4

PRP2,3 VB3,4

SBAR2,4

Analysis

Parsing

I1 saw2 her3 duck4

PRP0,1

VBD1,2

PRP$2,3 NN3,4

PRP2,3 VB3,4

SBAR2,4

Analysis

nodesO(Nn2)

O(Gn3) edges

Language Models Again•Language models are finite-state (i.e. regular).

•Our translation model is context-free.

•We can again compute full model via intersection.

•Result is also context-free.

•Bad news for context-free language models and context-free translation models...

•Context-free languages not closed under intersection.

•Computation is in PSPACE!

Language Models Again

•Basic DP strategy: nodes include category, span, and left and right language model context.

•While polynomial, this still tends to be too slow to do exactly.

•Various forms of pruning are generally used.

•Finding efficient algorithms is currently an area of very active research.

The Big Question

Where do the categories come from?

The Big Question

Where do the categories come from?Answer #1: there are no categories!

The Big Question

X → X1 X2 / X1 X2

X → X1 X2 / X2 X1

X → watashi wa / I X → hako wo / the boxX → akemasu / open

The Big Question

X → X1 X2 / X1 X2

X → X1 X2 / X2 X1

Keep order

The Big Question

X → X1 X2 / X1 X2

X → X1 X2 / X2 X1

Swap orderKeep order

The Big Question

X → X1 X2 / X1 X2

X → X1 X2 / X2 X1

The Big Question

X → X1 X2 / X1 X2

X → X1 X2 / X2 X1

Translate wordsor phrases

The Big Question

X Xwatashi wa

akemasuhako wo

open the box

Inversion Transduction Grammar

Inversion Transduction GrammarParsing is polynomial. We must be giving up something

in order to acheive polynomial complexity.

A B C D

B D A C

A B C D

B D A C

A B C D

B D A C

ITG cannot produce this kind of reordering.

A B C D

B D A C

ITG cannot produce this kind of reordering.Does this matter? Do such reorderings occur in real data?

Inversion Transduction Grammarje

pense que

, independamment

de notre parti

, nous

trouvonstous cela

inacceptable

that,Ibelieve,weall findunacceptable,regardlessofpoliticalparty

Inversion Transduction Grammarje

pense que

, independamment

de notre parti

, nous

trouvonstous cela

inacceptable

that,Ibelieve,weall findunacceptable,regardlessofpoliticalparty

YES! (but they’re very rare)

Hierarchical Phrase-Based Translation

the box

watashi wa akemasu

hako wo

X → X1 hako wo X2 / X1 open X2

X → hako wo / the boxX → akemasu / open

The Big Question

Where do the categories come from?Answer #2: from a parser.

The Big Question

Where do the categories come from?Answer #2: from a parser.

Syntax-based TranslationS S

NP VP NP VP

Are reorderings in real data consistent with isomorphisms on linguistic parse trees?

Syntax-based Translation

I saw her duck

PRP VBD PRP VB

Are reorderings in real data consistent with isomorphisms on linguistic parse trees?

yo la vi agacharse

Of course not.

I saw her duck

PRP VBD PRP VB

yo la vi agacharse

I saw her duck

PRP VBD PRP VB

yo la vi agacharse

Tree substitution grammar

I saw her duck

PRP VBD VB

yo la vi agacharse

weakly equivalent SCFG

I saw her duck

PRP VBD VB

yo la vi agacharse

S → PRP1 VP2 / PRP1 VP2

PRP → I / yo VP → VBD1 her VB2 / la VBD1 VB2

VBD → saw /vi VB → duck /agacharse

I saw her duck

PRP VBD VB

yo la vi agacharse

S → PRP1 VP2 / PRP1 VP2

PRP → I / yo VP → VBD1 her VB2 / la VBD1 VB2

VBD → saw /vi VB → duck /agacharse

Problem: we need a parser!

The Big Question

Where do the categories come from?Answer #3: they are automatically induced!

The Big Question

Where do the categories come from?Answer #3: they are automatically induced!

This is an area of active research.www.clsp.jhu.edu/workshops/ws10/groups/msgismt/

Another Big Question...

Where do the grammars come from?

Recap: Expectation Maximization

•Arbitrarily select a set of parameters (say, uniform).

•Calculate expected counts of the unseen events.

•Choose new parameters to maximize likelihood, using expected counts as proxy for observed counts.

•Iterate.

•Guaranteed that likelihood is monotonically nondecreasing.

Can we apply it to other models?

•Sure, why not?

•The derivation structure of each model is simply a latent variable.

•We simply apply EM to each model structure.

•Iterate.

BAD:Objective function is highly non-convex

•Iterate.

WORSE:Computing expectations from a phrase-based model, given a sentence pair, is NP-Complete(by reduction to SAT; DeNero & Klein, 2008)

•Iterate.

Computing expectations from an SCFGmodel, given a sentence pair, is at least O(n6)

Now What?

•Option #1: approximate expectations

•Restrict computation to some tractable subset of the alignment space (arbitrarily biased).

•Markov chain Monte Carlo (very slow).

Now What?•Option #2: change the problem definition

•We already know how to learn word-to-word translation models efficiently.

•Idea: learn word-to-word alignments, extract most probable alignment, then treat it as observed.

•Learn phrase translations consistent with word alignments.

•Decouples alignment from model learning -- is this a good thing?

Phrase Extraction

I open the box

watashi

akemasu

Phrase Extraction

I open the box

watashi

akemasu

watashi wa / I

Phrase Extraction

I open the box

watashi

akemasu

hako wo / the box

Phrase Extraction

I open the box

watashi

akemasu

hako wo akemasu / open the box

Phrase Extraction

I open the box

watashi

akemasu

Hierarchical Phrase Extraction

I open the box

watashi

akemasu

I open the box

watashi

akemasu

I open the box

watashi

akemasu

X1 akemasu / open X1

Syntactic Phrase Extraction

I open the box

watashi

akemasu

SNP VP

I open the box

watashi

akemasu

SNP VP

VP → hako wo akemasu/ open the box

I open the box

watashi

akemasu

SNP VP

VP → NP1 akemasu/open NP1

Summary

•Unsupervised learning over intractable models turns out to be a hard problem.

•Heuristic methods are widely used, but they offer no useful guarantees and are highly biased.

•Finding more elegant approximations is a topic of ongoing research.

Implementations

•Synchronous context-free translation models

•Moses -- www.statmt.org/moses

•cdec -- www.cdec-decoder.org

•Joshua -- www.cs.jhu.edu/~ccb/joshua

Datasets

•Proceedings of the European Parliament

•www.statmt.org/europarl

•Linguistic Data Consortium

•www.ldc.upenn.edu

Summary

•Many probabilistic translation models can be thought in terms of weighted (formal) languages.

•Dynamic programming is a common (though not universal!) decoding strategy.

•With these concepts in mind, you might be able to define models that capture other translation phenomena (e.g. morphosyntactics, semantics).

The Tower of Babel

Pieter Brueghel the Elder (1563)

2756 language pairs!

Statistical Machine Translation

Develop a statistical model of translation that can be learned from data and used to predict the correct

English translation of new Chinese sentences.

linguistics

algorithms

machine learning

formal language

theory info

Statistical Machine Translation

linguistics

algorithms

machine learning

formal language

theory info

Statistical Machine Translationregular &

context-free languages

Bayes’ rule,maximum likelihood,expectation

maximization

noisy channel model

syntax-basedmodels

dynamic programming,graphs & hypergraphs

The Data Deluge

•We are overwhelmed with data, but we can harness it to solve real problems.

•Formal tools help us model the data.

•Probabilistic tools help us learn models and make predictions.

•Algorithmic optimization methods make it all run.

•Tradeoffs: model expressivity vs. tractability.

We aren’t there yet!

•We still need:

•Better models of translation

•Based on linguistic insights

•Better approximations

•Better algorithms

Fred Jelinek18 November 1932 14 September 2010

Research in both ASR and MT continues. The statistical

approach is clearly dominant. The knowledge of linguists is added wherever it fits. And

although we have made significant progress, we are

very far from solving the problems.

Context-free translation models - mt-archive.info · Finite-State Models •Very simple models get...

Documents

Transcript of Context-free translation models - mt-archive.info · Finite-State Models •Very simple models get...

Data and Image Models - vis.berkeley.eduvis.berkeley.edu/courses/cs294-10-fa07/WWW/lectures-WWW/data/d… · Data Data models vs. Conceptual models Data models are low level descriptions

Petro Data Models

Database Models - Enterprise Architect UML modeling tool · Database Models 4 Data Modeling Overview 5 ... or UML Class models. Logical Data ModelsLogical data models add further

1 13. Spatiotemporal Databases Extreme Point Data Models Parametric Extreme Point Data Models Geometric Transformation Data Models Queries.

2e data models

Geographic Data Models

Other Data Models

Data Models - · PDF fileData Models ... ... Advantages/ Disadvantages •Advantages: •Simplicity •Data security •Data integrity

UN/CEFACT Reference Data Models (Semantic Models) · UN/CEFACT Reference Data Models (Semantic Models) Anticipating and Following Technological Trends in eBusiness Data Exchanges

Database system concept and Architecture. Data Models, Schemas, and Instance Data Models, Schemas, and Instance Data Models, Schemas, and Instance Data.

Nosql data models

Data base models

PPT Data Models

Different data models

Capturing Value from Data: Revenue Models for Data-Driven ...€¦ · Capturing Value from Data: Revenue Models for Data-Driven Services Ronny Schüritz ... models for data-driven

Data Modeling and Data Models The Importance of Data Models

Conversion between data models VectorRaster. Conversion between data models.

MODELS & DATA

Effective Data Models Data Management

GIS Data Models Components of a GIS Map Common Data Models.