Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March...
Transcript of Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March...
![Page 1: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/1.jpg)
Coreference Resolution with Knowledge
Haoruo Peng
March 20, 2015
1
![Page 2: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/2.jpg)
Coreference Problem
2
![Page 3: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/3.jpg)
Coreference Problem
3
![Page 4: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/4.jpg)
Online Demo
Please check
outhttp://cogcomp.cs.illinois.edu/page/demo_view/Coref
4
![Page 5: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/5.jpg)
General Framework
5
Jack threw the bags of Mary into the water
since he is angry with her.
![Page 6: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/6.jpg)
General Framework
Mention Detection
5
Jack threw the bags of Mary into the water
since he is angry with her.
![Page 7: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/7.jpg)
General Framework
Mention Detection
5
Jack threw the bags of Mary into the water
since he is angry with her.
![Page 8: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/8.jpg)
General Framework
Mention Detection
Pairwise Mention Scoring
5
Jack threw the bags of Mary into the water
since he is angry with her.
![Page 9: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/9.jpg)
General Framework
Mention Detection
Pairwise Mention Scoring
Goal: Coreferent Mentions have higher scores
5
Jack threw the bags of Mary into the water
since he is angry with her.
![Page 10: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/10.jpg)
General Framework
Mention Detection
Pairwise Mention Scoring
Goal: Coreferent Mentions have higher scores
5
Jack threw the bags of Mary into the water
since he is angry with her.
![Page 11: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/11.jpg)
General Framework
Mention Detection
Pairwise Mention Scoring
Goal: Coreferent Mentions have higher scores
Inference (ILP)
Best-Link
All-Link
5
Jack threw the bags of Mary into the water
since he is angry with her.
![Page 12: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/12.jpg)
ILP formulation of CR
Best-Link
All-Link
6
![Page 13: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/13.jpg)
General Framework
Learning (for Pairwise Mention Scores)
Structural SVM
Features:
Mention Types, String Relations, Semantic, Relative Location,
Anaphoricity(Learned), Aligned Modifiers, Memorization, etc.
7
![Page 14: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/14.jpg)
General Framework
Learning (for Pairwise Mention Scores)
Structural SVM
Features:
Mention Types, String Relations, Semantic, Relative Location,
Anaphoricity(Learned), Aligned Modifiers, Memorization, etc.
Evaluation
MUC
BCUB
CEAF_e
7
![Page 15: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/15.jpg)
General Framework
Learning (for Pairwise Mention Scores)
Structural SVM
Features:
Mention Types, String Relations, Semantic, Relative Location,
Anaphoricity(Learned), Aligned Modifiers, Memorization, etc.
Evaluation
MUC
BCUB
CEAF_e
IllinoisCoref
7
VS. Stanford Muti-pass Sieve System VS. Berkeley CR System
![Page 16: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/16.jpg)
Difficulties in CR
Hard Coreference Problems
[A bird] perched on the [limb] and [it] bent.
[Robert] is robbed by [Kevin], and [he] is arrested by police.
Gender / Plurality information cannot help
-> Requires Knowledge
8
![Page 17: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/17.jpg)
Part 1 Solving Hard Coreference Problems
9
![Page 18: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/18.jpg)
Hard Coreference Problems
Motivating Examples
10
![Page 19: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/19.jpg)
Hard Coreference Problems
Motivating Examples
Category 1
[A bird] perched on the [limb] and [it] bent.
[The bee] landed on [the flower] because [it] had pollen.
10
![Page 20: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/20.jpg)
Hard Coreference Problems
Motivating Examples
Category 1
[A bird] perched on the [limb] and [it] bent.
[The bee] landed on [the flower] because [it] had pollen.
Category 2
[Jack] is robbed by [Kevin], and [he] is arrested by police.
[Jim] was afraid of [Robert] because [he] gets scared around
new people.
10
![Page 21: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/21.jpg)
Hard Coreference Problems
Motivating Examples
Category 1
[A bird] perched on the [limb] and [it] bent.
[The bee] landed on [the flower] because [it] had pollen.
Category 2
[Jack] is robbed by [Kevin], and [he] is arrested by police.
[Jim] was afraid of [Robert] because [he] gets scared around
new people.
C
[Lakshman] asked [Vivan] to get him some ice cream because
[he] was hot.
10
![Page 22: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/22.jpg)
Predicate Schemas
Type 1
Type 2
11
![Page 23: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/23.jpg)
Predicate Schemas
Type 1
Type 2
11
![Page 24: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/24.jpg)
Predicate Schemas
Type 1
Type 2
11
Sub Obj
![Page 25: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/25.jpg)
Predicate Schemas
Type 1
(Cat1) [The bee] landed on [the flower] because [it] had pollen.
Type 2
11
Sub Obj
![Page 26: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/26.jpg)
Predicate Schemas
Type 1
(Cat1) [The bee] landed on [the flower] because [it] had pollen.
S(have(m=[the flower], a=[pollen])) >
S(have(m=[the bee], a=[pollen]))
Type 2
11
Sub Obj
![Page 27: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/27.jpg)
Predicate Schemas
Type 1
(Cat1) [The bee] landed on [the flower] because [it] had pollen.
S(have(m=[the flower], a=[pollen])) >
S(have(m=[the bee], a=[pollen]))
Type 2
11
Sub Obj
![Page 28: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/28.jpg)
Predicate Schemas
Type 1
(Cat1) [The bee] landed on [the flower] because [it] had pollen.
S(have(m=[the flower], a=[pollen])) >
S(have(m=[the bee], a=[pollen]))
Type 2
11
Sub Obj
![Page 29: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/29.jpg)
Predicate Schemas
Type 1
(Cat1) [The bee] landed on [the flower] because [it] had pollen.
S(have(m=[the flower], a=[pollen])) >
S(have(m=[the bee], a=[pollen]))
Type 2
11
Sub Obj
Shared Mention
![Page 30: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/30.jpg)
Predicate Schemas
Type 1
(Cat1) [The bee] landed on [the flower] because [it] had pollen.
S(have(m=[the flower], a=[pollen])) >
S(have(m=[the bee], a=[pollen]))
Type 2
(Cat2) [Jim] was afraid of [Robert] because [he] gets scared around
new people.
11
Sub Obj
Shared Mention
![Page 31: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/31.jpg)
Predicate Schemas
Type 1
(Cat1) [The bee] landed on [the flower] because [it] had pollen.
S(have(m=[the flower], a=[pollen])) >
S(have(m=[the bee], a=[pollen]))
Type 2
(Cat2) [Jim] was afraid of [Robert] because [he] gets scared around
new people.
S
S(be afraid of(a=*, m=*) | get scared around(m=*, a=*), because)
11
Sub Obj
Shared Mention
^
^
![Page 32: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/32.jpg)
Predicate Schemas
Possible variations for scoring function statistics.
12
![Page 33: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/33.jpg)
Predicate Schemas in Coreference
13
![Page 34: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/34.jpg)
Predicate Schemas in Coreference
Pairwise Mention Scoring Function
13
![Page 35: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/35.jpg)
Predicate Schemas in Coreference
Pairwise Mention Scoring Function
Scoring Function for Predicate Schemas
13
![Page 36: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/36.jpg)
Predicate Schemas in Coreference
Pairwise Mention Scoring Function
Scoring Function for Predicate Schemas
We can add scores of Predicate Schemas as Features
13
![Page 37: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/37.jpg)
Ways of Using Knowledge
Major Disadvantages of Using Knowledge as Features
Noise in Knowledge
Inexplicit Textual Inference
Alternative way
Using Knowledge as Constraints
14
![Page 38: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/38.jpg)
Using Knowledge as Constraints
15
![Page 39: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/39.jpg)
Using Knowledge as Constraints
Generating Constraints
15
![Page 40: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/40.jpg)
Using Knowledge as Constraints
Generating Constraints
ILP inference (Best-Link)
15
![Page 41: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/41.jpg)
Scores for Predicate Schemas
Multiple Sources
Gigaword
Wikipedia
Web Search
Polarity Information
16
![Page 42: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/42.jpg)
Scores for Predicate Schemas
Gigaword
Chunking + Dependency Parsing
=> predicate(subject, object)
=> Type 1 Predicate Schema
Heuristic Coreference
=> Type 2 Predicate Schema
17
![Page 43: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/43.jpg)
Scores for Predicate Schemas
Gigaword
Chunking + Dependency Parsing
=> predicate(subject, object)
=> Type 1 Predicate Schema
Heuristic Coreference
=> Type 2 Predicate Schema
Wikipedia
Entity Linking to ground on Wikipedia Entries (Disambiguation)
Gather Simple Statistics for 1) immediately after 2) immediately
before 3) after 4) before
=> Type 1 Predicate Schema
) (approximation)
17
![Page 44: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/44.jpg)
Scores for Predicate Schemas
Web Search
Google Query with Quote (counts)
“m predicate”, “m a”, “a m”, “m predicate a”
=> Type 1 Predicate Schema
18
![Page 45: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/45.jpg)
Scores for Predicate Schemas
Web Search
Google Query with Quote (counts)
“m predicate”, “m a”, “a m”, “m predicate a”
=> Type 1 Predicate Schema
Polarity Information
Polarity on predicates => Polarity on mentions
Negate polarity if mention is object
Negate polarity for polarity-reversing connective
+1 if polarities for mentions are the same
-1 if polarities for mentions are different
=> Type 2 Predicate Schema
18
![Page 46: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/46.jpg)
Recap
Things to consider for using knowledge in NLP
Knowledge Representation
Predicate Schema
Knowledge Inference
Features VS. Inference
Knowledge Acquisition
Multiple Sources
19
![Page 47: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/47.jpg)
Datasets
20
1http://www.hlt.utdallas.edu/~vince/data/emnlp12/
Winograd dataset1
[The bee] landed on [the flower] because [it] had pollen.
[The bee] landed on [the flower] because [it] wanted pollen.
![Page 48: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/48.jpg)
Datasets
20
1http://www.hlt.utdallas.edu/~vince/data/emnlp12/
Winograd dataset1
[The bee] landed on [the flower] because [it] had pollen.
[The bee] landed on [the flower] because [it] wanted pollen.
![Page 49: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/49.jpg)
Datasets
20
1http://www.hlt.utdallas.edu/~vince/data/emnlp12/
Winograd dataset1
[The bee] landed on [the flower] because [it] had pollen.
[The bee] landed on [the flower] because [it] wanted pollen.
[Jack] threw the bags of [John] into the water since [he]
mistakenly asked him to carry his bags.
![Page 50: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/50.jpg)
Datasets
20
1http://www.hlt.utdallas.edu/~vince/data/emnlp12/
Winograd dataset1
[The bee] landed on [the flower] because [it] had pollen.
[The bee] landed on [the flower] because [it] wanted pollen.
[Jack] threw the bags of [John] into the water since [he]
mistakenly asked him to carry his bags.
W
inoCoref dataset
![Page 51: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/51.jpg)
Evaluation on Hard Coreference
Dataset Winograd WinoCoref
Metric Precision AntePre
Illinois 51.48 68.37
IlliCons 53.26 74.32
Rahman and Ng (2012) 73.05 —–
KnowFeat 71.81 88.48
KnowCons 74.93 88.95
KnowComb 76.41 89.32
21
![Page 52: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/52.jpg)
Evaluation on Standard Coreference
22
![Page 53: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/53.jpg)
Analysis on Effects of Schemas
23
![Page 54: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/54.jpg)
Part 2 Profiler: Knowledge Schemas at Scale
24
![Page 55: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/55.jpg)
Goal
How to enlarge the Knowledge acquired from text
Data Volume
Schema Richness
Profiler
Demo: http://austen.cs.illinois.edu:60000/
25
![Page 56: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/56.jpg)
Motivation
26
![Page 57: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/57.jpg)
Enriched Schemas
27
![Page 58: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/58.jpg)
Enriched Schemas
28
![Page 59: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/59.jpg)
Enriched Schemas
29
![Page 60: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/60.jpg)
Implementation
30
![Page 61: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/61.jpg)
Effect of Wikification (Entity-Linking)
31
![Page 62: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/62.jpg)
Effect of Wikification (Entity-Linking)
32
![Page 63: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/63.jpg)
Knowledge Visualization
33
![Page 64: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/64.jpg)
Knowledge Visualization
34
![Page 65: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/65.jpg)
Publications
[1] Solving Hard Coreference Problems. Haoruo Peng*,
Daniel Khashabi* and Dan Roth. NAACL 2015.
[2] A Joint Framework for Mention Head Detection and
Coreference Resolution. Submitted to ACL 2015.
[3] Profiler: Knowledge Schemas at Scale. Submitted to
Transactions of ACL 2015 .
35
![Page 66: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/66.jpg)
Future Directions
The use of world knowledge in NLP tasks
Knowledge Representation (schemas)
Is co-occurrence information enough?
Knowledge Inference
Sparsity Issues
Knowledge Acquisition
Which sources to choose?
Interpolation /
Tasks beyond CR (CR can be seen as a subset of AI-complete
problems)
Outlier Detection for Singleton Mentions
36
![Page 67: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/67.jpg)
Collaborators
Daniel Khashabi Zhiye Fei Kaiwei Chang
Prof. Dan Roth
37
![Page 68: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/68.jpg)
38
Thank You !
![Page 69: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/69.jpg)
Difficulties in CR
Hard Coreference Problems
Ex.1 [A bird] perched on the [limb] and [it] bent.
Ex.2 [Robert] is robbed by [Kevin], and [he] is arrested by police.
Gender / Plurality information cannot help
-> Requires Knowledge
39
![Page 70: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/70.jpg)
Difficulties in CR
Hard Coreference Problems
Ex.1 [A bird] perched on the [limb] and [it] bent.
Ex.2 [Robert] is robbed by [Kevin], and [he] is arrested by police.
Gender / Plurality information cannot help
-> Requires Knowledge
39
![Page 71: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/71.jpg)
Difficulties in CR
Hard Coreference Problems
Ex.1 [A bird] perched on the [limb] and [it] bent.
Ex.2 [Robert] is robbed by [Kevin], and [he] is arrested by police.
Gender / Plurality information cannot help
-> Requires Knowledge
39
![Page 72: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/72.jpg)
Difficulties in CR
Hard Coreference Problems
Ex.1 [A bird] perched on the [limb] and [it] bent.
Ex.2 [Robert] is robbed by [Kevin], and [he] is arrested by police.
Gender / Plurality information cannot help
-> Requires Knowledge
Performance Gaps
-> Requires Better Mention Detection
39
System Dataset Gold Predicted Gap
Illinois CoNLL-12 77.05 60.00 17.05
Berkeley CoNLL-11 76.68 60.42 16.26
Stanford ACE-04 81.05 70.33 10.72
![Page 73: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/73.jpg)
A Joint Framework for Mention Head Detection and Coreference Resolution
Goal: Improve CR on predicted mentions (End-to-End)
Solution:
Traditional: MD -> Coref
Our paper: Mention Head -> Joint Coref -> Head to Mention
Joint Learning / Inference Step
Add decision variables to decide whether to choose a head or not
Joint Coref is able to reject some mention head candidates
Results
40
[Multinational companies investing in [China]] had become so angry that [they] recently set up an anti-piracy league to pressure [the [Chinese] government] to take action. [Domestic manufacturers, [who] are also suffering], launched a similar body this month.
Dataset Illinois Baseline Our Paper
ACE-04 68.27 68.27 71.20
CoNLL-12 60.00 61.71 63.01
![Page 74: Coreference Resolution with Knowledgedanielkhashabi.com/files/2015_Winograd_naacl/DAIS... · March 20, 2015 1 . Coreference Problem 2 . Coreference Problem 3 . Online Demo](https://reader036.fdocuments.us/reader036/viewer/2022071218/60539ee00ab8127da731acc4/html5/thumbnails/74.jpg)
ILP formulation of CR
Best-Link with Knowledge Constraints
Best-Link with Joint Mention Detection
41