Machine Learning GroupDepartment of Computer Sciences

University of Texas at Austin

Learning Language Semantics from Ambiguous Supervision

Rohit J. Kate Raymond J. Mooney

2

Semantic Parsing• Involves learning language semantics to transform natural

language (NL) sentences into computer executable complete meaning representations (MRs) for some application

• Geoquery: An example database query application

Which rivers run through the states bordering Texas?

Queryanswer(traverse(next_to(stateid(‘texas’))))

Semantic Parsing

Arkansas, Canadian, Cimarron, Gila, Mississippi, Rio Grande …

Answer

3

Learning for Semantic Parsing

• Learning for semantic parsing consists of inducing a semantic parser from training data which can map novel sentences into their meaning representations

• Many accurate learning systems for semantic parsing have been recently developed: [Ge & Mooney, 2005], [Zettlemoyer & Collins, 2005], [Wong & Mooney, 2006], [Kate & Mooney, 2006], [Nguyen, Shimazu & Phan, 2006]

4

Unambiguous Supervision for Learning Semantic Parsers

• The training data for semantic parsing consists of hundreds of natural language sentences unambiguously paired with their meaning representations

5

Unambiguous Supervision for Learning Semantic Parsers

• The training data for semantic parsing consists of hundreds of natural language sentences unambiguously paired with their meaning representations

Which rivers run through the states bordering Texas?

answer(traverse(next_to(stateid(‘texas’))))

What is the lowest point of the state with the largest area?

answer(lowest(place(loc(largest_one(area(state(all)))))))

What is the largest city in states that border California?

answer(largest(city(loc(next_to(stateid( 'california'))))))

……

6

Shortcomings of Unambiguous Supervision

• It requires considerable human effort to annotate each sentence with its correct meaning representation

• Does not model the type of supervision children receive when they are learning a language– Children are not taught meanings of individual

sentences

– They learn to identify the correct meaning of a sentence from several meanings possible in their perceptual context

7

???

“Mary is on the phone”

8

Ambiguous Supervision for Learning Semantic Parsers

• A computer system simultaneously exposed to perceptual contexts and natural language utterances should be able to learn the underlying language semantics

• We consider ambiguous training data of sentences associated with multiple potential meaning representations– Siskind (1996) uses this type “referentially uncertain” training data to

learn meanings of words

• Capturing meaning representations from perceptual contexts is a difficult unsolved problem – Our system directly works with symbolic meaning representations

9

“Mary is on the phone”

???

10

“Mary is on the phone”???

11

Ironing(Mommy, Shirt)

“Mary is on the phone”???

12

Ironing(Mommy, Shirt)

Working(Sister, Computer)

“Mary is on the phone”???

13

Ironing(Mommy, Shirt)

Working(Sister, Computer)

Carrying(Daddy, Bag)

“Mary is on the phone”???

14

Ironing(Mommy, Shirt)

Working(Sister, Computer)

Carrying(Daddy, Bag)

Talking(Mary, Phone)

Sitting(Mary, Chair)

“Mary is on the phone”

Ambiguous Training Example

???

15

Ironing(Mommy, Shirt)

Working(Sister, Computer)

Talking(Mary, Phone)

Sitting(Mary, Chair)

“Mommy is ironing shirt”

Next Ambiguous Training Example

???

16

Ambiguous Supervision for Learning Semantic Parsers contd.

• Our model of ambiguous supervision corresponds to the type of data that will be gathered from a temporal sequence of perceptual contexts with occasional language commentary

• We assume each sentence has exactly one meaning in a perceptual context

• Each meaning is associated with at most one sentence in a perceptual context

17

Sample Ambiguous Corpus

Daisy gave the clock to the mouse.

Mommy saw that Mary gave the hammer to the dog.

The dog broke the box.

John gave the bag to the mouse.

The dog threw the ball.

ate(mouse, orange)

gave(daisy, clock, mouse)

ate(dog, apple)

saw(mother, gave(mary, dog, hammer))

broke(dog, box)

gave(woman, toy, mouse)

gave(john, bag, mouse)

threw(dog, ball)

runs(dog)

saw(john, walks(man, dog))

Forms a bipartite graph

18

Rest of the Talk

• Brief background on KRISP, the semantic parsing learning system for unambiguous supervision

• KRISPER: Extended system to handle ambiguous supervision

• Corpus construction

• Experiments

19

KRISP: Semantic Parser Learner for Unambiguous Supervision

• KRISP: Kernel-based Robust Interpretation for Semantic Parsing [Kate & Mooney 2006]

• Takes NL sentences unambiguously paired with their MRs as training data

• Treats the formal MR language grammar’s productions as semantic concepts

• Trains an SVM classifier for each production with string subsequence kernel [Lodhi et al. 2002]

20

MR: answer(traverse(next_to(stateid(‘texas’))))Parse tree of MR:

Productions: ANSWER answer(RIVER) RIVER TRAVERSE(STATE)

STATE NEXT_TO(STATE) TRAVERSE traverse NEXT_TO next_to STATEID ‘texas’

ANSWER

answer

STATE

RIVER

STATE

NEXT_TO

TRAVERSE

STATEID

stateid ‘texas’

next_to

traverse

Meaning Representation Language

ANSWER answer(RIVER)

RIVER TRAVERSE(STATE)

TRAVERSE traverse STATE NEXT_TO(STATE)

NEXT_TO next_to STATE STATEID

STATEID ‘texas’

21

Semantic Parsing by KRISP

• SVM classifier for each production gives the probability that a substring represents the semantic concept of the production

Which rivers run through the states bordering Texas?

NEXT_TO next_to 0.02 0.01

NEXT_TO next_toNEXT_TO next_to 0.95

22

Semantic Parsing by KRISP

• SVM classifier for each production gives the probability that a substring represents the semantic concept of the production

Which rivers run through the states bordering Texas?

TRAVERSE traverseTRAVERSE traverse 0.210.91

23

Semantic Parsing by KRISP

• Semantic parsing is done by finding the most probable derivation of the sentence [Kate & Mooney

2006]

Which rivers run through the states bordering Texas?

ANSWER answer(RIVER)

RIVER TRAVERSE(STATE)

TRAVERSE traverse STATE NEXT_TO(STATE)

NEXT_TO next_to STATE STATEID

STATEID ‘texas’

0.91

0.95

0.89

0.92

0.81

0.98

0.99

Probability of the derivation is the product of the probabilities at the nodes.

24

Semantic Parsing by KRISP

• Given a sentence and a meaning representation, KRISP can also find the probability that it is the correct meaning representation for the sentence

25

KRISPER: KRISP with EM-like Retraining

• Extension of KRISP that learns from ambiguous supervision

• Uses an iterative EM-like method to gradually converge on a correct meaning for each sentence

26

KRISPER’s Training Algorithm

Daisy gave the clock to the mouse.

Mommy saw that Mary gave the hammer to the dog.

The dog broke the box.

John gave the bag to the mouse.

The dog threw the ball.

ate(mouse, orange)

gave(daisy, clock, mouse)

ate(dog, apple)

saw(mother, gave(mary, dog, hammer))

broke(dog, box)

gave(woman, toy, mouse)

gave(john, bag, mouse)

threw(dog, ball)

runs(dog)

saw(john, walks(man, dog))

1. Assume every possible meaning for a sentence is correct

27

KRISPER’s Training Algorithm

Daisy gave the clock to the mouse.

Mommy saw that Mary gave the hammer to the dog.

The dog broke the box.

John gave the bag to the mouse.

The dog threw the ball.

ate(mouse, orange)

gave(daisy, clock, mouse)

ate(dog, apple)

saw(mother, gave(mary, dog, hammer))

broke(dog, box)

gave(woman, toy, mouse)

gave(john, bag, mouse)

threw(dog, ball)

runs(dog)

saw(john, walks(man, dog))

1. Assume every possible meaning for a sentence is correct

28

KRISPER’s Training Algorithm contd.

Daisy gave the clock to the mouse.

Mommy saw that Mary gave the hammer to the dog.

The dog broke the box.

John gave the bag to the mouse.

The dog threw the ball.

ate(mouse, orange)

gave(daisy, clock, mouse)

ate(dog, apple)

saw(mother, gave(mary, dog, hammer))

broke(dog, box)

gave(woman, toy, mouse)

gave(john, bag, mouse)

threw(dog, ball)

runs(dog)

saw(john, walks(man, dog))

2. Resulting NL-MR pairs are weighted and given to KRISP

1/2

1/2

1/41/4

1/41/4

1/5 1/51/5

1/51/5

1/3 1/31/3

1/31/3

1/3

29

KRISPER’s Training Algorithm contd.

Daisy gave the clock to the mouse.

Mommy saw that Mary gave the hammer to the dog.

The dog broke the box.

John gave the bag to the mouse.

The dog threw the ball.

ate(mouse, orange)

gave(daisy, clock, mouse)

ate(dog, apple)

saw(mother, gave(mary, dog, hammer))

broke(dog, box)

gave(woman, toy, mouse)

gave(john, bag, mouse)

threw(dog, ball)

runs(dog)

saw(john, walks(man, dog))

3. Estimate the confidence of each NL-MR pair using the resulting parser

30

KRISPER’s Training Algorithm contd.

Daisy gave the clock to the mouse.

Mommy saw that Mary gave the hammer to the dog.

The dog broke the box.

John gave the bag to the mouse.

The dog threw the ball.

ate(mouse, orange)

gave(daisy, clock, mouse)

ate(dog, apple)

saw(mother, gave(mary, dog, hammer))

broke(dog, box)

gave(woman, toy, mouse)

gave(john, bag, mouse)

threw(dog, ball)

runs(dog)

saw(john, walks(man, dog))

3. Estimate the confidence of each NL-MR pair using the resulting parser

0.92

0.11

0.320.88

0.220.24

0.71 0.180.85

0.140.95

0.24 0.890.33

0.970.81

0.34

31

KRISPER’s Training Algorithm contd.

Daisy gave the clock to the mouse.

Mommy saw that Mary gave the hammer to the dog.

The dog broke the box.

John gave the bag to the mouse.

The dog threw the ball.

ate(mouse, orange)

gave(daisy, clock, mouse)

ate(dog, apple)

saw(mother, gave(mary, dog, hammer))

broke(dog, box)

gave(woman, toy, mouse)

gave(john, bag, mouse)

threw(dog, ball)

runs(dog)

saw(john, walks(man, dog))

4. Use maximum weighted matching on a bipartite graph to find the best NL-MR pairs [Munkres, 1957]

0.92

0.11

0.320.88

0.220.24

0.180.85

0.24 0.890.33

0.970.81

0.34

0.71

0.950.14

32

KRISPER’s Training Algorithm contd.

Daisy gave the clock to the mouse.

Mommy saw that Mary gave the hammer to the dog.

The dog broke the box.

John gave the bag to the mouse.

The dog threw the ball.

ate(mouse, orange)

gave(daisy, clock, mouse)

ate(dog, apple)

saw(mother, gave(mary, dog, hammer))

broke(dog, box)

gave(woman, toy, mouse)

gave(john, bag, mouse)

threw(dog, ball)

runs(dog)

saw(john, walks(man, dog))

4. Use maximum weighted matching on a bipartite graph to find the best NL-MR pairs [Munkres, 1957]

0.92

0.11

0.320.88

0.220.24

0.180.85

0.24 0.890.33

0.970.81

0.34

0.71

0.950.14

33

KRISPER’s Training Algorithm contd.

Daisy gave the clock to the mouse.

Mommy saw that Mary gave the hammer to the dog.

The dog broke the box.

John gave the bag to the mouse.

The dog threw the ball.

ate(mouse, orange)

gave(daisy, clock, mouse)

ate(dog, apple)

saw(mother, gave(mary, dog, hammer))

broke(dog, box)

gave(woman, toy, mouse)

gave(john, bag, mouse)

threw(dog, ball)

runs(dog)

saw(john, walks(man, dog))

5. Give the best pairs to KRISP in the next iteration, continue till converges

34

Corpus Construction

• There is no real-world ambiguous corpus yet available for semantic parsing to our knowledge

• We artificially obfuscated the real-world unambiguous corpus by adding extra distracter MRs to each training pair (Ambig-Geoquery)

• We also created an artificial ambiguous corpus (Ambig-ChildWorld) which more accurately models real-world ambiguities in which potential candidate MRs are often related

35

Ambig-Geoquery Corpus

NL

NL

NL

NL

NL

MR

MR

MR

MR

MR

Start with the unambiguous Geoquery corpus

36

Ambig-Geoquery Corpus

NL

NL

NL

NL

NL

MR

MR

MR

MR

MR

Insert 0 to random MRs from the corpus between each pair

MR

MR

MR

MRMRMR

MR

MR

MR

MR

37

Ambig-Geoquery Corpus

NL

NL

NL

NL

NL

MR

MR

MR

MR

MR

Form a window of width from 0 to in either direction for each NL sentence

MR

MR

MR

MRMRMR

MR

MR

MR

MR

38

Ambig-Geoquery Corpus

NL

NL

NL

NL

NL

MR

MR

MR

MR

MR

Form the ambiguous corpus

MR

MR

MR

MRMRMR

MR

MR

MR

MR

39

Ambig-ChildWorld Corpus

• Although Ambig-Geoquery corpus uses real-world NL-MR pairs, it does not model relatedness between potential MRs for each sentence, common in perceptual contexts

• Constructed a synchronous grammar [Aho &

Ullman, 1972] to simultaneously generate artificial NL-MR pairs

• Uses 15 verbs and 37 nouns (people, animals, things), MRs are in predicate logic without quantifiers

40

Ambig-ChildWorld Corpus contd.

• Different perceptual contexts were modeled by choosing subsets of productions of the synchronous grammar

• This leads to subsets of verbs and nouns (e.g. only Mommy, Daddy, Mary) causing more relatedness among potential MRs

• For each such perceptual context, data was generated in a way similar to Ambig-Geoquery corpus

41

Ambiguity in Corpora

• Three levels of ambiguity were created by varying parameters and

MRs per NL 1 2 3 4 5 6 7

Level 1 25% 50% 25%

Level 2 11% 22% 34% 22% 11%

Level 3 6% 13% 19% 26% 18% 12% 6%

42

Methodology

• Performed 10-fold cross validation

• Metrics:

• Measured best F-measure across the precision-recall curve obtained using output confidence thresholds

MRsoutputcompletewithsentencestestofNumber

MRscorrectofNumberPrecision =

€

Recall = Number of correct MRs

Number of test sentences

€

F − measure = 2* Pr ecision *Recall

Pr ecision + Recall

43

Results on Ambig-Geoquery Corpus

0

10

20

30

40

50

60

70

80

90

100

225 450 675 900

Number of training examples

Best F-measure

No ambiguity

Level 1 ambiguity

Level 2 ambiguity

Level 3 ambiguity

44

Results on Ambig-ChildWorld Corpus

0

10

20

30

40

50

60

70

80

90

100

225 450 675 900

Number of training examples

Best F-measure

No ambiguity

Level 1 ambiguity

Level 2 ambiguity

Level 3 ambiguity

45

Future Work

• Construct a real-world ambiguous corpus and test this approach

• Combine this system with a vision-based system that extracts MRs from perceptual contexts

46

Conclusions

• We presented the problem of learning language semantics from ambiguous supervision

• This form of supervision is more representative of natural training environment for a language learning system

• We presented an approach that learns from ambiguous supervision by iteratively re-training a system for unambiguous supervision

• Experimental results on two artificial corpora showed that this approach is able to cope with ambiguities to learn accurate semantic parsers

47

Thank you!

Questions??