1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.
-
Upload
destinee-loll -
Category
Documents
-
view
220 -
download
1
Transcript of 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.
![Page 1: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/1.jpg)
1
Ido Dagan Bar-Ilan University, Israel
Natural Language Inference
in Natural Language Terms
![Page 2: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/2.jpg)
BIU NLP lab - Acknowledgments
Chaya Liebeskind
![Page 3: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/3.jpg)
3
Let’s look at state-of-the-art common practice…
![Page 4: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/4.jpg)
4
NLP Applications need morphology So… we use an available morphological “engine”! Namely – a morphological analyzer
Input: word (in context) Output: possible morphological analyses
(scored/ranked)
But there are lot’s of morphological phenomena to model
Yes, but they are all embedded within the morphological analyzer
![Page 5: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/5.jpg)
5
NLP Applications need syntax So… we use an available syntactic “engine”! Namely – a parser
Input: sentence Output: possible parse trees (scored/ranked)
But there are lot’s of syntactic phenomena to model Yes, but they are all embedded within the parser
Frameworks which embed modeling under a coherent task
![Page 6: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/6.jpg)
6
NLP Applications need semantics So…. what do we do?
Use NER, WordNet, SRL, WSD, statistical similarities, syntactic matching, detect negations, etc, etc, etc.
Assemble and implement bits and pieces
Scattered & redundant application-dependent research
Focus & critical mass lacking, s l o w progress
Can we have a generic semantic inference framework? Semantic interpretation into logic?
But hardly adopted (why?), so I put it aside till later…
![Page 7: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/7.jpg)
7
What is inference?From dictionary.com:
inferring:to derive by reasoning; conclude or judge from premises or evidence.
reasoning:the process of forming conclusions, judgments, or inferences from facts or premises.
![Page 8: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/8.jpg)
8
What is inference?From dictionary.com:
inferring:to derive by reasoning; conclude or judge from premises or evidence.
reasoning:the process of forming conclusions, judgments, or inferences from facts or premises.
![Page 9: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/9.jpg)
9
Textual Inference
Inferring new textual expressions from given ones
Captures two types of inference:1. Inferences about the “extra-linguistic” world
it rained yesterday => it was wet yesterday
2. Inferences about language variability I bought a watch => I purchased a watch
No definite boundary between the two
![Page 10: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/10.jpg)
10
Textual Entailment – a definition capturing textual inference A directional relation between two text
fragments: Text (t) and Hypothesis (h):
t entails h (th) if humans reading t will infer that h is most likely true
Operational (applied) definition: Human gold standard Assuming common background knowledge
Language & world knowledge
![Page 11: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/11.jpg)
Motivation: Inference in Applications
QA:Question: What affects blood pressure?
“Salt causes an increase in blood pressure”
IE: X purchase YIBM Coremetrics
Google reMail
Yahoo Overture
IR:Query: symptoms of IBS
“IBS is characterized by vomiting ”
![Page 12: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/12.jpg)
Entailment inMulti-document Summarization
Obama gave a speech last night in the Israeli lobby
conference
Obama gave a speech last night in the Israeli lobby
conference
In his speech at the American Israel Public Affairs
Committee yesterday, the president challenged …
In his speech at the American Israel Public Affairs
Committee yesterday, the president challenged …
Barack Obama’s AIPAC address yesterday ...
Barack Obama’s AIPAC address yesterday ...
![Page 13: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/13.jpg)
13
Appeal of textual entailment definition Became a prominent view on textual inference
RTE 1-7; 1950 hits in Google Scholar
Much more concrete than: “paraphrase”
bi-directional entailment / equivalence “partial highly-covering entailment”
“similarity” – very vague (non-scientific?) notion
Additional textual inference types may be defined But they should be defined, reasonably precisely
![Page 14: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/14.jpg)
14
Evaluation: PASCAL RTE Challenges
TEXT HYPOTHESIS TASK ENTAIL-MENT
1Regan attended a ceremony in Washington to commemorate the landings in Normandy.
Washington is located inNormandy. IE False
2 Google files for its long awaited IPO. Google goes public. IR True
3
…: a shootout at the Guadalajara airport in May, 1993, that killed Cardinal Juan Jesus Posadas Ocampo and six others.
Cardinal Juan Jesus Posadas Ocampo died in 1993.
QA True
4
The SPD got just 21.5% of the votein the European Parliament elections,while the conservative opposition partiespolled 44.5%.
The SPD is defeated bythe opposition parties. IE True
• Created utilizing (or simulating) reductions from real systems’ output
![Page 15: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/15.jpg)
15
Initial use of RTE systems in applications QA
Harabagiu & Hickl, ACL-06 Answer Validation Exercise (AVE) at CLEF QallMe (FBK-irst, Magnini et al.)
Relation extraction Romano et al., EACL-06
Educational applications Nielsen et al., ACL-08 education workshop, SemEval/RTE-8
Summarization Harabagiu et al. 2007, Information Processing and Management
MT evaluation and paraphrasing for MT (two ACL-2009 papers)
![Page 16: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/16.jpg)
16
The Textual Entailment Taskvs. Classical Approach to InferenceMeaning
Representation Language
Natural Language
Classical logic-based approach: - Natural language isn’t suitable for conducting inference
- Too vague, ambiguous, … - We need to invent artificial languages (logics) that support inference
Inference
Interpretation approach
Textual Entailment
![Page 17: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/17.jpg)
17
Textual inferences in practice –are based on NL representations Recognizing target expressions (QA, IE, …)
Lexical substitutions and similarity Matching syntactic and predicate patterns,
semantic roles Machine learning based on linguistic features
Co-reference chains for discourse objects
“Natural” name classes (vs. abstract classes) “football player”, “coffee producer”, …
![Page 18: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/18.jpg)
18
NL-based approach
How should computers infer?
MeaningRepresentation
Language
Natural Language
Alternative language-based approach:
- Perform many inferences over natural language representations
- May resort to extra-linguistic representations/inference when needed
Inference
Interpretation approach
Textual Entailment
![Page 19: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/19.jpg)
19
Appeal of NL representations
If people think in NL, why shouldn’t computers?...
Saves the need of logic interpretation And the need to invent (and agree on!) logics NL representations are consensual and obtainable
Easier to acquire inference knowledge Particularly with unsupervised learning methods
A great challenge – more later…
![Page 20: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/20.jpg)
Desiderata1. Develop principled & practical inference over NL
representations Analogous to principled “logics” (learning based) Most current applied inferences are ad-hoc
(in RTE or application-specific)
2. Develop methods for acquiring vast inference knowledge Represented in language structures
3. Develop generic platforms/engines that implement both of the above
* Other fields as role models: MT, parsing – similar investment needed!
20
![Page 21: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/21.jpg)
21
Principled Learning-based Inference Mechanisms
- over language structures
![Page 22: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/22.jpg)
Knowledge and Tree-Edits in Learnable Entailment Proofs
Asher Stern and Ido Dagan(earlier partial version by Roy Bar-Haim)
Download at: http://www.cs.biu.ac.il/~nlp/downloads/biutee
BIUTEE
![Page 23: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/23.jpg)
23
Transformation-based Inference
Sequence of transformations (A proof)
Tree-Edits Complete proofs – by limited pre-defined set of
operations Estimate confidence in each operation
Knowledge based Entailment Rules Arbitrary knowledge-based transformations Formalize many types of knowledge
T = T0 → T
1 → T
2 → ... → T
n = H
![Page 24: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/24.jpg)
24
Transformation based RTE - Example
T = T0 → T
1 → T
2 → ... → T
n = H
Text: The boy was located by the police.
Hypothesis: Eventually, the police found the child.
![Page 25: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/25.jpg)
Transformation based RTE - Example
T = T0 → T
1 → T
2 → ... → T
n = H
Text: The boy was located by the police.
The police located the boy.
The police found the boy.
The police found the child.
Hypothesis: Eventually, the police found the child.
25
![Page 26: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/26.jpg)
Transformation based RTE - Example
T = T0 → T
1 → T
2 → ... → T
n = H
26
![Page 27: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/27.jpg)
27
BIUTEE’s Inference Formalism
Analogy to logic proof systems:
Parse Trees Propositions
Tree transformation/generation Inference Steps
Sequence of generated trees: T … Ti … H Proof
![Page 28: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/28.jpg)
28
BIUTEE Goals
Rely on Entailment Rules Supported by many types of knowledge
Tree Edits Allow complete proofs
BIUTEE Integrates the benefits of both Estimate confidence of both
![Page 29: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/29.jpg)
29
Challenges / System Components
1. generate linguistically motivated complete proofs?
2. estimate proof confidence?
3. find the best proof?
4. learn the model parameters?
How to…
![Page 30: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/30.jpg)
1. Generate linguistically motivated complete proofs
30
![Page 31: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/31.jpg)
Knowledge-based Entailment Rules
boy child
Generic Syntactic
Lexical Syntactic
Lexical
Bar-Haim et al. 2007. Semantic inference at the lexical-syntactic level .
![Page 32: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/32.jpg)
32
Extended Tree Edits (On The Fly Operations)
Predefined custom tree edits Insert node on the fly Move node / move sub-tree on the fly Flip part of speech …
Heuristically capture linguistic phenomena Operation definition Features – to estimate confidence
![Page 33: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/33.jpg)
Proof over Parse Trees - Example
T = T0 → T
1 → T
2 → ... → T
n = H
Text: The boy was located by the police.
Passive to active
The police located the boy.
X locate Y X find Y
The police found the boy.
Boy child
The police found the child.
Tree-edit insertion
Hypothesis: Eventually, the police found the child.
33
![Page 34: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/34.jpg)
34
Co-reference Substitution For co-referring subtrees S1 , S2:
Copy source tree containing S1 while replacing it with S2
My brother is a musician. He plays the drums.
beverb
ROOTi
musiciannoun
brothernoun
subj
mynoun
gen
adet
det
predplayverb
drumnoun
thedet
obj
ROOTi
henoun
det
subjplayverb
drumnoun
thedet
obj
ROOTi
det
subj
My brother plays the drums.
brothernoun
mynoun
gen
![Page 35: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/35.jpg)
2. Estimate proof confidence
35
![Page 36: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/36.jpg)
36
Cost based Model (Variant of Raina et al., 2005) Define operation cost
Represent each operation as a feature vector Cost is linear combination of feature values
Define proof cost as the sum of the operations’ costs
Classify: entailment if and only if proof cost is lower than a threshold
![Page 37: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/37.jpg)
Feature vector representation Define operation feature value Represent each operation as a feature vectorFeatures (Insert-Named-Entity, Insert-Verb, … , WordNet, Lin, DIRT, …)
The police located the boy.DIRT: X locate Y X find Y (score = 0.9)
The police found the boy.
)0,0,…,0.257,…,0((0 ,0,…,0,…,0)Feature vector that
represents the operation 37
An operation
A downward function of score
![Page 38: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/38.jpg)
38
Cost based Model
Define operation cost– Cost is standard linear combination of feature values
Cost = weight-vector * feature-vector Weight-vector is learned automatically
)())(( ofwofC Tw
![Page 39: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/39.jpg)
Confidence Model
Define operation cost Represent each operation as a feature vector
Define proof cost as the sum of the operations’ costs
)()()()(11
PfwofwoCPC Tn
ii
Tn
iiww
Cost of proofWeight vector
Vector represents the proof.
Define)()(
1
Pfofn
ii
![Page 40: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/40.jpg)
40
Feature vector representation - example
T = T0 → T
1 → T
2 → ... → T
n = H
)0,0,..…….………………,1,0(
)0,0……..………,0.457,..,0,0(
)0,0…..,0.5,..…….……….,0,0(
)0,0,1,....…..….……..……,0,0(
)0,0,1..0.5…..0.457…....,1,0(
+
+
+
=
Text: The boy was located by the police.
Passive to active
The police located the boy.
X locate Y X find Y
The police found the boy.
Boy child
The police found the child.
Insertion on the fly
Hypothesis: Eventually, the police found the child.
![Page 41: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/41.jpg)
Cost based Model Define operation cost
Represent each operation as a feature vector Define proof cost as the sum of the
operations’ costs Classify: “entailing” if and only if proof cost is
smaller than a threshold
bPfwT )(
41Learn
![Page 42: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/42.jpg)
3. Find the best proof
42
![Page 43: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/43.jpg)
43
Search the best proof
Proof #1Proof #2Proof #3Proof #4
T - H
![Page 44: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/44.jpg)
44
Search the best proof
• Need to consider the “best” proof for the positive pairs• “Best Proof” = proof with lowest cost
‒ Assuming a weight vector is given• Search space exponential – AI-style search (ACL-12)
‒ Gradient-based evaluation function‒ Local lookahead for “complex” operations
Proof #1Proof #2Proof #3Proof #4
T HProof #1Proof #2Proof #3Proof #4
T H
![Page 45: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/45.jpg)
4. Learn model parameters
45
![Page 46: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/46.jpg)
46
Learning
Goal: Learn parameters (w, b) Use a linear learning algorithm
logistic regression
![Page 47: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/47.jpg)
47
Inference vs. Learning
Training samples
Vector representation
Learning algorithm
w,bBest
Proofs
Feature extraction
Feature extraction
![Page 48: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/48.jpg)
48
Inference vs. Learning
Training samples
Vector representation
Learning algorithm
w,bBest
Proofs
Feature extraction
![Page 49: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/49.jpg)
49
Iterative Learning Scheme
Training samples
Vector representation
Learning algorithm
w,bBest
Proofs
1 .W=reasonable guess
2 .Find the best proofs
3 .Learn new w and b
4 .Repeat to step 2
![Page 50: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/50.jpg)
50
Summary- System Components
1. Generate syntactically motivated complete proofs?
Entailment rules On the fly operations (Extended Tree Edit Operations)
2. Estimate proof validity? Confidence Model
3. Find the best proof? Novel search Algorithm
4. Learn the model parameters? Iterative Learning Scheme
How to
![Page 51: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/51.jpg)
Results RTE 1-5
51
System RTE-1 RTE-2 RTE-3 RTE-5
Raina et al. 2005 57.0
Harmeling, 2009 56.39 57.88
Wang and Manning, 2010 63.0 61.10
Bar-Haim et al., 2007 61.12 63.80
Mehdad and Magnini, 2009 58.62 59.87 62.4 60.2
Our System 57.13 61.63 67.13 63.50
Text: Hypothesis:
Text: Hypothesis:
Evaluation by accuracy – comparison with transformation-based systems
![Page 52: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/52.jpg)
52
Results RTE 6
RTE 6 (F1%)
Base line (Use IR top-5 relevance) 34.63
Median (2010) 36.14
Best (2010) 48.01
Our system 49.54
Natural distribution of entailmentsEvaluation by Recall / Precision / F1
![Page 53: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/53.jpg)
53
Conclusions – The BIUTEE Inference Engine Inference as proof over parse trees
Natural to incorporate many inference types Results - close to best or best on RTEs
Open Source Configurable Extensible Visual tracing Support
![Page 54: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/54.jpg)
54
Adding extra-linguistic inferences Some tasks may benefit from extra-linguistic
“expert” inferences Temporal / arithmetic / spatial reasoning / …
2 soldiers and a civilian => 3 people
Need to integrate with primary inference over language structures “Expert” may detect on the fly inferences that would
bridge text and hypothesis, Interleaved within tree-generation process
![Page 55: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/55.jpg)
55
Slide from Inderjeet Mani
![Page 56: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/56.jpg)
56
Related Research Lines RTE via tree edits
Learning edit costs, compute Tree-Edit-Distance Mehdad & Magnini 2009; Heilman & Smith 2010;
Wang & Manning 2010
Text-to-text generation Cf. ACL-2011 workshop, Smith’s invited talk
Paraphrasing – recognition, generation Richer discourse-level inferences
Mirkin at al. 2010 (merging, bridging) Implicit argument detection
Gerber 2010, SemEval task 2010, Ruppenhofer et al. 2011
Recovering implicit relations Nakov & Kozareva 2011
Natural logic MacCartney & Manning 2009
![Page 57: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/57.jpg)
Lexical Textual Inference [Eyal Shnarch]
Complex systems use parser
Lexical inference rules link terms from T to H Lexical rules come from lexical resources H is inferred from T iff all its terms are inferred
Improves state-of-the-artprincipled probabilistic model lexical textual inference
In the Battle of
Waterloo, 18 Jun
1815, the French
army, led by
Napoleon, was
crushed.
in Belgium Napoleon was defeated
PLIS - Probabilistic Lexical Inference System
1st or 2nd order co-
occurrence
Text Hypothesis
57/34
![Page 58: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/58.jpg)
Probabilistic model – forward inference
Improves state-of-the-artprincipled probabilistic model lexical textual inference
T
H which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushed
)( HTP
knowledge integration
term-levelsentence-level
)( 3hTP )( 1hTP
t1 t2 t3 t4 t5 t6
h1 h2 h3
)( 2hTP
annotations are available at
sentence-level only
PLIS - Probabilistic Lexical Inference System
x1 x2 x3
58/34
![Page 59: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/59.jpg)
Backward witnesses model
PLIS - Probabilistic Lexical Inference System
which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushedt1 t2 t3 t4 t5 t6
h1 h2 h3
x1 x2 x3
AND
y
𝑃 (𝑊 ( h𝑖 )|𝑥 𝑖=1)= ∏𝑤 ∈𝑊 (h𝑖)
𝜃𝑤 ⋅ ∏𝑤∉𝑊 (h𝑖)
(1−𝜃𝑤)
𝜂0 ≝ 𝑃 (𝑥 𝑖=1∨𝑦=0)𝜂1 ≝ 𝑃 (𝑥𝑖=1∨𝑦=1)
𝜃𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥 𝑖=1)
𝜏𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥𝑖=0) 𝑃 (𝑊 ( h𝑖 )|𝑥 𝑖=0 )= ∏𝑤∈𝑊 (h𝑖)
𝜏𝑤 ⋅ ∏𝑤∉𝑊 (h𝑖)
(1−𝜏𝑤)
Likelihood
Improves state-of-the-artprincipled probabilistic model lexical textual inference
59/34
![Page 60: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/60.jpg)
60
Acquiring Inference Knowledge
- over language structures
![Page 61: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/61.jpg)
61
Knowledge acquisition sources Learning from corpora Mining human-oriented knowledge resources
Wikipedia, dictionary definitions Computational NLP resources
WN, FrameNet, NOMLEX, … Manual knowledge engineering
Recent Mechanical Turk potential
![Page 62: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/62.jpg)
Distributional similarity (Symmetric) Most similar words for food (Lin, 1998)
Symmetric measure often identifies “sister” terms
meat clothing water sugar
beverage foodstuff coffee material
goods textile meal chemical
medicine fruit tobacco equipment
drink feed fuel rice
![Page 63: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/63.jpg)
63
Directional similarity – Feature Inclusion Kotlerman et al. (2009)
Most directionally-similar words for food :
foodstuff ration blanket margarine
food product drinking water soup dessert
food company wheat flour biscuit cookie
noodle grocery sweetener sauce
canned food beverage meat ingredient
feed snack agribusiness meal
salad dressing dairy product diet vegetable
bread hamburger medicine vegetable oil
food aid chocolate food supply herb
drink seafood fruit juice milk
![Page 64: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/64.jpg)
Extraction from Wikipedia
• Be-complement• TopAll-nouns • BottomAll-nouns
• Redirectvarious terms to canonical title
• Be-complement• Redirect• Parenthesis• Link
64
![Page 65: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/65.jpg)
65
Learning Entailment-rule Graphs
• Berant et al. series of works – increasing scalability:ACL-2010, ACL-2011 (best paper by student), ACL-2012
• Example target hierarchy (medical domain):
![Page 66: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/66.jpg)
66
Global Optimization of Graph Edges
}1,0{
1.),(
0.),(
1.,,
),(maxargˆ
uv
uv
uv
uwvwuv
uv
vu
I
IPOSvu
INEGvu
IIIVwvu
IvufG
• Integer Linear Program• Optimize global edge scores under transitivity and
other constraints
u
w
v
66
11
0
![Page 67: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/67.jpg)
67
Syntactic-driven Entailments Active-passive transformations Recover relative clause arguments Extract conjuncts Appositions …
TruthTeller: annotate truth for predicates and clauses Positive: John called(+) Mary. Negative: John forgot to call(−) Mary. Unknown: John wanted to call(?) Mary.
Constructed via human linguistic engineering May be combined with automatic learning
![Page 68: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/68.jpg)
68
Mechanical Turk & Community Knowledge-engineering Validating automatically-learned rules
Generating paraphrases/entailments Zeichner et al., ACL-2012
Potential for community contribution Stipulating domain knowledge in NL
![Page 69: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/69.jpg)
Inference & ambiguityH The US accepts a large number of foreigners
every year
T The US welcomes hundreds of thousands of aliens yearly
If it’s any consolation, dear, our alien
abduction insurance is finally going to pay off
alien foreignerwelcome accept
![Page 70: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/70.jpg)
Context matching
Context Matching generalizes sense matching Does aliens in T match the meaning of outer-space ? Does ‘children acquire English’ match
X acquire Y X learn Y?
Contextual Preferences A generic context validation framework for entailment rules
Szpektor & Dagan, ACL-08 Classification-based approach (Mirkin et al., TextInfer 2011)
Match hypothesis, rules and text only in suitable contexts An alternative to explicit WSD
![Page 71: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/71.jpg)
71
BIUTEE Demo
![Page 72: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/72.jpg)
72
EXCITEMENT: towards Textual-inference Platform - Open source & community
![Page 73: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/73.jpg)
73
A Textual Inference Platform
Starting with BIUTEE, moving to EXCITEMENT Goal: build MOSES-like environment
Incorporate partners’ inference systems Addressing two types of research communities:
Applications which can benefit from textual inference Technologies which can improve inference technology
Partners: Academic: FBK, Heidelberg, DFKI, Bar-Ilan Industriacl: NICE (Israel), AlmaWave (Italy), OMQ (Germany)
![Page 74: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/74.jpg)
74
Future: Extended Operation modes Recognition: recognize entailment given T/H pair
Validation in applications
Search: given H and corpus/doc, find all entailing texts Multi-document summarization (RTE-5 pilot & RTE-6) QA, IR, IE against corpus/doc Use entailment knowledge to generate expanded queries
Generation: given text, generate all entailed statements Paraphrase generation for MT Unsupervised IE – generate “canonical” propositions
Functionality extensions Include variables in hypothesis (perform extraction - IE, QA, …) Partial entailment: identify entailments of parts of h …
![Page 75: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/75.jpg)
Entailment-based Text Exploration
with Application to the Health-care Domain
Meni Adler, Jonathan Berant, Ido Dagan
ACL 2012 Demo
![Page 76: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/76.jpg)
Motivation
![Page 77: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/77.jpg)
Goal
Guide users through concrete facts in unfamiliar domains
Common exploration approaches – via concepts/categories “what are the documents talking about?”
Our approach – exploration of statements/facts “what do the documents actually say?”
![Page 78: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/78.jpg)
Entailment-Based Exploration Approach
Key Points
Collect statements/facts about the target concept Open IE (Etzioni et al.) How to organize it?
Organize statements (propositions) by entailment hierarchies
![Page 79: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/79.jpg)
Statement Level
Propositions Predicate and Arguments
![Page 80: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/80.jpg)
Statement Level
Propositions Predicate and Arguments
Improper lighting causes eyestrain headaches
X – cause – Y
![Page 81: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/81.jpg)
Entailment Hierarchies
Entailment Relation
Improper lighting causes eyestrain headaches Improper lighting result in eyestrain headaches
Entailment RuleX – cause – Y
X – result in – Y
![Page 82: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/82.jpg)
Entailment Hierarchies
Entailment Graph for predicates
associate headache with X
X trigger headache X is a source of headacheX cause headache
X result in headache
![Page 83: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/83.jpg)
Entailment Hierarchies
Concept Taxonomy (for arguments)
Medical
Diseases
Cancer
Asthama
Drugs
Aspirin
Flexeril
![Page 84: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/84.jpg)
System Resources
Extracted Propositions)TextRunner, ReVerb(… ,
chocolate - trigger- Headacheacetaminophen - reduce –headachetea – treat – headache…
CorpusPredicate Entailment Graph
Concept Taxonomy (optional)
![Page 85: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/85.jpg)
Exploration Scheme
1 .User looks for information on headache
2 .User drills down the entailment hierarchy for items that trigger headache
3 .Focus on Food which triggers headache, particularly chocolate
4 .The retrieved document indicates that chocolate may trigger headaches
![Page 86: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/86.jpg)
Exploration Scheme
2 .Drill down the entailment hierarchy for items that are produced by the brain
3 .User focuses on chemicals that are produced by the brain
1 .User looks for information on brain
![Page 87: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/87.jpg)
Conclusions – Exploration System A novel powerful exploration paradigm
Organize extracted statements by entailment hierarchies
Enables fast acquisition of knowledge by users
Current Work Learning a general entailment graph for the
health-care domain Prototype available
Investigating appropriate evaluation methods
![Page 88: 1 Ido Dagan Bar-Ilan University, Israel Natural Language Inference in Natural Language Terms.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551c45415503467b488b4cb8/html5/thumbnails/88.jpg)
88
Overall Takeout Time to develop textual inference
Generic, applied, principled
Proposal: Base core inference on language-based representations
Parse trees, co-references, lexical contexts, … Extra-linguistic/logical inference for specific suitable cases
Breakthrough potential – current and future applications
It’s a long-term endeavor, but it’s here!
http://www.cs.biu.ac.il/~nlp/downloads/biutee