Natural Language Models and Interfacesivan-titov.org/teaching/nlmi-15/lecture-11.pdf(bruja verde,...

Natural Language Models and Interfaces��lecture 11

Ivan Titov

Institute for Logic, Language and Computation

Today

2

}  Machine translation:

}  IBM 1 and HMM models

}  Unsupervised Induction of Semantic Representations

}  Topics for the exam

IBM Candide Project [Brown et al 93]

French Broken English

English

French/English Bilingual Text

English Text

Statistical Analysis Statistical Analysis

J’ ai si faim

What hunger have I, Hungry I am so, I am so hungry, Have me that hunger …

I am so hungry

Mathematical Formulation

J’ ai si faim I am so hungry

Translation Model P(f | e)

Language Model P(e)

Decoding algorithm argmaxe P(e) · P(f | e)

Given source sentence f:

argmaxe P(e | f) =

argmaxe P(f | e) · P(e) / P(f) = by Bayes Rule

argmaxe P(f | e) · P(e) P(f) same for all e

French Broken English

English

The Classic Translation Model Word Substitution/Permutation [Brown et al., 1993]

Mary did not slap the green witch

Mary not slap slap slap the green witch

Maria no dió una bofetada a la bruja verde

Mary not slap slap slap NULL the green witch

Maria no dió una bofetada a la verde bruja

Trainable

Process model of translation:

n(3|slap) 50k entries

d(j|i) 2500 entries

P-Null 1 entry

t(la|the) 25m entries

Let's do it more formally

Unsupervised EM Training

… la maison … la maison bleue … la fleur … … the house … the blue house … the flower …

All P(french-word | english-word) equally likely



“la” and “the” observed to co-occur frequently, so P(la | the) is increased.



“maison” co-occurs with both “the” and “house”, but P(maison | house) can be raised without limit, to 1.0,

while P(maison | the) is limited because of “la”

(pigeonhole principle)

e f P(f | e) national

nationale 0.47 national 0.42 nationaux 0.05 nationales 0.03

the

le 0.50 la 0.21 les 0.16 l’ 0.09 ce 0.02 cette 0.01

farmers

agriculteurs 0.44 les 0.42 cultivateurs 0.05 producteurs 0.02

new French sentence f

P(f | e) · P(e) à score for e

Translation Model

Language Model

w1 w2 P(w2 | w1) of

the 0.13 a 0.09 another 0.01 some 0.01

hong

kong 0.98 said 0.01 stated 0.01

potential translation e

P(f | e) P(e)

Beam Search Decoding [Brown et al US Patent #5,477,451]

1st English word

2nd English word

3rd English word

4th English word

start end

Each partial translation hypothesis contains: - Last English word chosen + source words covered by it - Next-to-last English word chosen - Entire coverage vector (so far) of source sentence - Language model and translation model scores (so far)

all source words

covered

[Jelinek 69; Och, Ueffing, and Ney, 01]

Phrases

SOURCE TARGET

phrases phrases

How do you translate “real estate” into French? real estate real number dance number dance card memory card memory stick …

words words

syntax syntax

logical form

interlingua

logical form

Phrase-Based Statistical MT

•  Foreign input segmented into phrases –  “phrase” just means “word sequence”

•  Each phrase is probabilistically translated into English –  P(to the conference | zur Konferenz) –  P(into the meeting | zur Konferenz)

•  Phrases are probabilistically re-ordered See [Koehn et al, 2003] for an overview. – More details now

Morgen fliege ich nach Kanada zur Konferenz

Tomorrow I will fly to the conference In Canada

How to Learn the Phrase Translation Table?

•  One method: “alignment templates” [Och et al 99]

•  Start with word alignment

•  Collect all phrase pairs that are consistent with the word alignment

How do we compute word alignments?



Word Alignment Induced Phrases




(Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green)




(Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap) (bruja verde, green witch)




(Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap) (Maria no, Mary did not) (no dió una bofetada, did not slap), (dió una bofetada a la, slap the) (bruja verde, green witch) (Maria no dió una bofetada, Mary did not slap) (a la bruja verde, the green witch) … (Maria no dió una bofetada a la bruja verde, Mary did not slap the green witch)

Phrase Pair Probabilities

•  A certain phrase pair (f-f-f, e-e-e) may appear many times across the bilingual corpus.

•  No EM training

•  Just relative frequency:

count(f-f-f, e-e-e) P(f-f-f | e-e-e) = ----------------------- count(e-e-e)

Phrase-Based MT •  It was the best way to do Statistical MT until very recently •  Now syntax starts play the role

•  Like CFGs, but production have two right hand sides – Source side – Target side – Related through linked non-terminal symbols

•  E.g. VP → <V[1] NP[2],NP[2] V[1]> •  One-to-one correspondence •  Productions applied in parallel to both sides to

linked non-terminals

Synchronous CFGs [Chiang, 2005]

Today

21





Representing events and their participants

22

}  A semantic frame [Fillmore 1968] is a conceptual structure describing a situation, object, or event along with associated properties and participants

}  Example: OPENING frame

Jack opened the lock with a paper clip

Semantic Roles (aka Frame Elements):

OPENER – an initiator/doer in the event [Who?]

OPENED - an affected entity [to Whom / to What?]

INSTRUMENT – the entity manipulated to accomplish the goal

Other roles for CLOSURE/OPENING frame: BENEFICIARY, FASTENER, DEGREE,

CIRCUMSTANCES, MANIPULATOR, PORTAL, …

Can we induce this without labeled data?

Lansky left Australia to study the piano at the Royal College of Music

Semantic frames and roles }  A semantic frame [Fillmore 1968] is a conceptual structure describing a

situation, object, or event along with associated properties and participants




AGENT – an initiator/doer in the event [Who?]

PATIENT - an affected entity [to Whom / to What?]


Other roles for CLOSURE/OPENING frame: BENEFICIARY, FASTENER, DEGREE, CIRCUMSTANCES,

MANIPULATOR, PORTAL, …

1


Student

SubjectInstituition

EducationDeparting

Object

Source

Purpose











2


Student

SubjectInstituition

EducationDeparting

Object

Source

Purpose











2

Can we induce this without labeled data?

effect

affected

Frame: likelihood

frame

predicate (i.e. its lemma)

role 1

(non-terminal, head word) for role 1

tend

likelihood

affected

(NP, investments)

[Thompson et al, 2003]

Recap: last time

Induction of Frame-Semantic Information

27

}  The semantic induction task involves 3 sub-tasks

}  Construction of a transformed syntactic dependency graph (~ argument identification)

gave Peter the Great build wooden fortified castlean order to a

gave Peter the Great build wooden fortified castlean order

Person Request BuildingsBeing_ProtectedConstruction to

Material a


28



}  Induction of frames (and clusters of arguments)


Person Request

Speaker Message

Created Entity

BuildingsBeing_ProtectedConstruction to

Material

Material

a

Type


29



}  Induction of frames (and clusters of arguments)

}  Role Induction

We model these sub-tasks jointly within a probabilistic model

We will refer to labels as semantic classes (both frames and just clusters of arguments)

The (Simplified) Model

30

GenSemClass(croot

)

croot

⇠ �root

for each sentence :Draw semantic class for root


31

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :

for each role t = 1, . . . , T :

s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :

Request


32

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :


s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :

Draw synt/lex realization

gave an order

Request

Like in the syntactic DOP model


33

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :


s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :


At least one argument

gave an order

Request

Speaker


34

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :


s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :



Draw first argument

GenArgument(c, t)

ac,t ⇠ �c,t

c0c,t ⇠ �c,t

GenSemClass(c0c,t)

gave an order

Request

Speaker


35

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :


s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :



Draw first argument

GenArgument(c, t)

ac,t ⇠ �c,t

c0c,t ⇠ �c,t

GenSemClass(c0c,t)

Draw argument key

gave an order

Request

SpeakerACTIVE:LEFT:SBJ

Syntactic property of the argument


36

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :


s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :



Draw first argument

GenArgument(c, t)

ac,t ⇠ �c,t

c0c,t ⇠ �c,t

GenSemClass(c0c,t)

Draw argument key

Draw semantic class for arg

gave an order

Person Request



37

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :


s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :



Draw first argument

GenArgument(c, t)

ac,t ⇠ �c,t

c0c,t ⇠ �c,t

GenSemClass(c0c,t)

Draw argument key


Recurse

gave Peter the Great an order

Person Request



38

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :


s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :



Draw first argument

GenArgument(c, t)

ac,t ⇠ �c,t

c0c,t ⇠ �c,t

GenSemClass(c0c,t)

Draw argument key


Recurse Continue generation

gave Peter the Great an order

Person Request



39

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :


s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :



Draw first argument

GenArgument(c, t)

ac,t ⇠ �c,t

c0c,t ⇠ �c,t

GenSemClass(c0c,t)

Draw argument key



Draw more arguments

gave Peter the Great buildan order

Person Request

Speaker Message

Constr. to

ACTIVE:LEFT:SBJ ACTIVE:RIGHT:OBJ


40

GenSemClass(croot

)

croot

⇠ �root


while [n ⇠ �+c,t] = 1 :


s ⇠ �c

GenArgument(c, t)

GenArgument(c, t)

GenSemClass(c)

if [n ⇠ �c,t] = 1 :



Draw first argument

GenArgument(c, t)

ac,t ⇠ �c,t

c0c,t ⇠ �c,t

GenSemClass(c0c,t)

Draw argument key



Draw more arguments

gave Peter the Great build fortified castlean order

Person Request

Speaker Message

Created Entity

BuildingsProtectedConstr. to a

Type

ACTIVE:LEFT:SBJ ACTIVE:RIGHT:OBJ

ACTIVE:RIGHT:OBJ

-:LEFT:NMOD

Inducing Semantics

41

}  Given a (large) collection of sentences annotated with (transformed) syntactic dependencies

gave Peter the Great build wooden fortified castlean order to a

woreMary an evening dress from Cardin

…

{xi}ni=1


}  We want to induce its semantics , i.e. its segmentation and clustering


Person Request

Speaker Message

Created Entity

BuildingsBeing_ProtectedConstruction to

Material

Material

a

Type


Wearer

Wearing

Clothing Creator

Style

GarmentPerson Occasion Brand

Inducing Semantics

42

…

{xi}ni=1

{mi}ni=1


1121 621

9 3

11

33332395 to

78

24

a

4


12

573

5 21

7

8971121 445 1621


}  We want to induce its semantics , i.e. its segmentation and clustering

Inducing Semantics

43

…

{xi}ni=1

{mi}ni=1

Can be induced with a form of EM algorithm (though this is not quite what we have done)

Semantic classes (frames)

44

From a collection of biomedical texts:

Class Variations

1 motif, sequence, regulatory element, response element, element, dna sequence

2 donor, individual, subject

3 important, essential, critical

4 dose, concentration

5 activation, transcriptional activation, transactivation

6 b cell, t lymphocyte, thymocyte, b lymphocyte, t cell, t-cell line, human lymphocyte, t-lymphocyte

7 indicate, reveal, document, suggest, demonstrate

8 augment, abolish, inhibit, convert, cause, abrogate, modulate, block, decrease, reduce, diminish, suppress, up-regulate, impair, reverse, enhance

9 confirm, assess, examine, study, evaluate, test, resolve, determine, investigate

10 nf-kappab, nf-kappa b, nfkappab, nf-kb

“cause change position on a scale” frame

Blood cells

Today





Overview

46

}  Noisy channel paradigm (and applications)

}  Language models:

}  handling rare words and noise

}  Sequence labeling problems:

}  POS-tagging, spell-checking applications and semantic role labeling (SRL)

}  Estimation algorithms

}  Decoding (labeling) methods: Viterbi (dynamic programming)

}  Parsing problems (trees)

}  Syntactic parsing and touched on semantics (probabilistic CCGs)

}  Modeling: PCFGs and grammar transforms (including DOP)

}  Decoding (parsing) algorithms: CKY (dynamic programming)

}  Sequence-to-sequence problems

}  Machine translation: word, phrase and syntactic models, …

Material for the Exam (March 24th)

47

}  CFG and PCFGs }  incl. tree probability; estimation from a treebank

}  recap smoothing

}  CNF form, binarization (CNF transform)

}  CKY algorithm for CFG and PCFG

}  Incl. understanding the data structure (the chart), understanding the unary closures both for the probabilistic case and not, time complexity of the algorithm

}  Make sure that given a grammar and a sentence, you can fill a chart

}  Ideas about weaknesses of PCFGs

}  Remember the PP-attachment and coordination examples


48

}  Structural annotation: vertical and horizontal Markovization }  Effects on grammar size, on parsing speed, accuracy, …

}  Lexicalization: how to get rules, basic understanding of estimation, time complexity

}  Computing the probability of a word sequence with PCFGs

}  "Sum product" – inside probabilities


49

}  DOP estimation (recall the basic corpus example)

}  HMMs and PCFGs as general tool:

}  Semantic role labeling problem: the basic sequence model

}  CCGs (only the application combinator, no composition and no type raising): an idea how CKY can be applied to the semantic parsing problem


50

}  Machine translation

}  Word models: IBM model1, HMM model (from today)

}  An idea about higher order IBM model (see slides from lecture 9 – modeling valency)

}  The basic phrase-based model: how do we obtain phrases? How do we define probabilistic model? (including from today)

}  Intuition about estimation (an idea of the EM algorithm but not formal derivation)

}  Decoding: what represents the state in the lattice?

}  An idea about synchronous CFGs


51

}  Not at the exam:

}  Semantics

}  Type-raising and composition for CCGs

}  unsupervised induction of SRL (i.e. today)

}  Cognitive aspects

}  History

Natural Language Models and Interfacesivan-titov.org/teaching/nlmi-15/lecture-11.pdf(bruja verde,...

Documents

Transcript of Natural Language Models and Interfacesivan-titov.org/teaching/nlmi-15/lecture-11.pdf(bruja verde,...