Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science...
-
Upload
chrystal-harrison -
Category
Documents
-
view
220 -
download
2
Transcript of Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science...
![Page 1: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/1.jpg)
Page 1
Relation Alignment for Textual Entailment Recognition
Department of Computer ScienceUniversity of Illinois at Urbana-Champaign
Mark Sammons, V.G.Vinod Vydiswaran, Tim Vieira, Nikhil Johri, Ming-Wei Chang, Dan Goldwasser, Vivek Srikumar, Gourab Kundu, Yuancheng Tu, Kevin Small, Joshua Rule, Quang Do, Dan Roth
![Page 2: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/2.jpg)
Cognitive Computation Group approach to RTE Problem statement: Given a corpus of entailment
pairs (T,H) Task requires us to ‘explain’ the content of the Hypothesis
(H) using content in the Text (T) (plus some background knowledge).
If all elements in H can be explained, label is ‘YES’; otherwise, ‘NO’.
Goals: Attack the TE problem using divide-and-conquer strategy
Plug-and-play architecture Avoid pipeline architecture where possible
Apply Machine Learning with “justifiable” features Assumptions:
Each element of H (word, phrase, predicate) is explained by one element of T (may require representing T differently).
Semantics is (largely) compositional: we can determine entailment for element s of (T, H) individually, and integrate them to get the global label for the entailment pair.
Page 2
![Page 3: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/3.jpg)
CCG TE System: Approach
Develop and apply Similarity Metrics for semantic elements annotated by standard NLP resources Named Entity, Numerical Quantity, Semantic Role,
Shallow Parse chunks; others in development Allow encapsulation and modular incorporation of
background knowledge Compare two constituents (generated from a specific
analysis source); return a real number [-1, 1](oppositional – not comparable – similar)
Use similarity metrics to resolve local entailment decisions
No guarantees that outputs are scaled – e.g. 0.8 may mean ‘low similarity’ for NE, ‘nearly identical’ for lexical metric
Page 3
![Page 4: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/4.jpg)
John Smith bought three cakes and two oranges
John bought two oranges
Alignment in RTE: Lexical Level
Page 4
But which decisions are relevant to global label? Many possible comparisons, when all constituents considered
Use alignment as basis for inference; select ‘relevant’ local decisions
![Page 5: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/5.jpg)
System Overview
Page 5
Aligner
Multi-View rep.
Alignments
Graph
LabelFeature Extractor, Classifier
i i kijiji THeTHeTHe
m)),(),,((.),(
11
j
ji THeI 1)],([
![Page 6: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/6.jpg)
Outline: Alignment in Textual Entailment
What is Alignment?
Models of Alignment/Previous work
CCG RTE
Results and Conclusions
Page 6
![Page 7: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/7.jpg)
John Smith bought three cakes and two oranges
John bought two oranges
Alignment in RTE: Lexical Level
Page 7
Alignment: a mapping from elements in the Hypothesis to elements in the Text
![Page 8: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/8.jpg)
Alignment is Useful for Machine Learning in RTE
Machine Learning approaches provide much-needed robustness for NLP tasks
RTE data sets are small, given complexity of problem
Global, 2- or 3-class label on each pair We would like to resolve entailment by
combining local decisions (e.g. word-level, phrase level); but *which* decisions?
Alignment can be used to select a subset of the many possible comparisons, and thereby augments global label with (proxy for) finer-grained structure; can be used…
…to determine active features …to generate labels for local classifiers
Page 8
![Page 9: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/9.jpg)
John Smith bought three cakes and two oranges
John bought two oranges
John Smith bought three cakes and two oranges
John bought three oranges
Alignment in RTE: Lexical Level
Page 9
![Page 10: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/10.jpg)
John told his counselor he needed a job.
A counselor needed a job.
Alignment in RTE: Parse level
Page 10
John told his counselor John needed a job.
A counselor needed a job.
his counselor
![Page 11: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/11.jpg)
Models of Alignment in RTE (Previous Work)
Alignment as entailment discriminator: e.g. Zanzotto et al. (2006) Use lexical matching + parse tree similarity measure to
build ‘best’ match graph for entailment pairs Allows robustness against parse errors, minor variations in
structure Use inter-pair graph similarity measure to determine
entailment “Alignment As Entailment”
Alignment as feature selector: e.g. de Marneffe et al. (2007), Hickl et al. (2006) Infer alignment over words/phrases in entailment pairs Extract features from aligned constituents, train entailment
classifier “Alignment as Filter”
Page 11
![Page 12: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/12.jpg)
Approaches to “Alignment As Filter”
Page 12
![Page 13: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/13.jpg)
John Smith said he bought three cakes and two oranges
John bought two oranges
John Smith said Jane bought three cakes and two oranges
John bought three oranges
Shallow Alignment as Focus Of Attention
Pick a “good” shallow alignment (possibly, highest scoring matches)
Use these to query deeper structure
Page 13
![Page 14: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/14.jpg)
John Smith said he bought three cakes and two oranges
John bought two oranges
John Smith said Jane bought three cakes and two oranges
John bought three oranges
Using Structure as Focus Of Attention
Find best structural match; base entailment results on results of shallow comparison resources
Page 14
![Page 15: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/15.jpg)
Some Questions
It seems we have a choice: Find a shallow alignment, and then rely on subsequent
stages to model the structural constraint.
OR Use the structural constraint to inform the shallow
alignment.
Is one of the two approaches preferable? Mistakes in shallow alignment problem inferring
structural constraint. There may be many ‘good’ shallow alignments…
Errors in deep structure problem selecting correct local decision
Other preprocessing errors – e.g. Coreference – will propagate either way
Can we avoid these problems?
Page 15
![Page 16: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/16.jpg)
CCG RTE
Page 16
![Page 17: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/17.jpg)
Multiple alignments at multiple granularities
Intuition: exploit differences/agreements between different views of the entailment pair; avoid canonization
Accommodates analysis at different granularities
Resources with comparable scores can compete with each other – pick the “best” e.g. Words, Multi-word Expressions, Phrasal Verbs
Unscaled resources occupy different alignments (SRL, NE)
Metrics can return negative numbers; use magnitude in alignments, preserve negative edge label May be useful for contradiction features
Page 17
![Page 18: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/18.jpg)
Page 18
T: John Smith said Jane bought three cakes and two oranges
H: Jane bought three oranges
Janebuy
three cakes two oranges
Jane buy three oranges
John Smith
say
Multiple Alignments for RTE
[#:3][unit: orange]
[#: 3] [unit: cake] [#:2] [unit:orange]
[PER] John Smith [PER] Jane
[PER] Jane
NE
NUM
SRL
SRL
NUM
NE
![Page 19: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/19.jpg)
Learning from Multiple Alignments
Extract features based on individual alignments Can use high-precision, low-recall resources as
filter features Typical match features within alignments – e.g.
proportion of tokens matched
Extract features based on agreement, disagreement between different alignments E.g. Predicate-Argument, Numerical Quantities
Allows graceful degradation if some resources are unreliable; learner assigns low weights to corresponding features
Page 19
![Page 20: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/20.jpg)
Multiple Alignments ctd.
Model each alignment as optimization problem Penalize distant mappings of neighboring
constituents in H, T (proxy for deep structure – favor chunk alignment)
Constraints: each token in H can be covered exactly once by an aligned constituent; edge scores must account for number of constituents covered
Solve by brute-force search
Page 20
i i kijiji THeTHeTHe
m)),(),,((.),(
11
j
ji THeI 1)],([
![Page 21: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/21.jpg)
Feature Extraction
Main types of features: Features assessing quality of alignment in a given view Features assessing agreement between views
Quality of Alignment features: Proportion of constituents matched in Word, NE, SRL
views “Distortion” of match pattern
Agreement features: Proportion of token alignments agreeing with SRL
constituent alignments Negation of predicate in SRL relation match
Extension: Using Coreference: Augment SRL predicates: add arguments using Coref
chains Introduces inter-sentence structure
Page 21
![Page 22: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/22.jpg)
Results and Conclusions
Page 22
![Page 23: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/23.jpg)
* Submitted runs had ~60 buggy alignments in dev test; results using non-buggy alignments shown here
Results
Page 23
Corpus
System
Baseline
No NE*
Basic NE
No WN
All* All + Coref
RTE5 Dev
0.628 0.640 0.623 0.647 0.648 0.663
RTE5 Test
0.600 0.629 0.633 0.603 0.644 0.666
![Page 24: Page 1 Relation Alignment for Textual Entailment Recognition Department of Computer Science University of Illinois at Urbana-Champaign Mark Sammons, V.G.Vinod.](https://reader036.fdocuments.us/reader036/viewer/2022062716/56649de55503460f94add14e/html5/thumbnails/24.jpg)
Conclusions Good improvement over ‘smart’ lexical baseline;
not (yet) close to best system (73.5%) Some further performance gain may be possible
from tuning distance penalty in alignment optimization, additional features
Reasonable behavior in ablation Coreference-based structure helps – improves
shallow semantic match Possible utility from avoiding canonization
(pipeline) Multiple Alignment + Feature Extraction offers
flexible system, avoids shallowdeep, deepshallow problems
Alignment + Metric approach seems promising as general framework for RTE
Page 24