Identifying the tradeoffs in textual entailment: deep representation versus shallow entailment Randy...

Identifying the tradeoffs in textual entailment: deep representation versus shallow entailment

Randy GoebelAlberta Innovates Centre for Machine Learning

Department of Computing ScienceUniversity of AlbertaEdmonton, Alberta

[email protected]

NII Shonan Meeting Seminar 057, November 27, 2014.

Fuji-san


BIRS


Science or Engineering?

• Google’s translation team employs no linguistics

• Does the challenge of textual entailment include scientific challenges?

• Is there an objective function/measure for “optimal” textual entailment?


Outline

• A simple textual entailment framework• Landmarks in the history of deep

representation• Building models of language entailment

from annotated corpora• Explanation based on multiple entailment

models• Summary


Simple Textual Entailment

Tom loves all women.

X woman(X) loves(Tom,X)

Does Tom love Sally?

Loves(Tom,Sally)rep

rese

nta

tion

expre

ssib

ility

entailment precision

A

B


Landmarks of deep representation …

• WVO Quine’s relational versus notional• Russell, Hilbert, Robinson descriptive

terms• Montague’s formal philosophy• Steedman’s induction of semantic parsing


Quine’s Relational versus Notional


Quine’s Relational versus Notional

“John wants a sloop.”

relational notional

X.wants(John,X) & sloop(X)

X.wants(John,X) & sloop(X) & name(X, “JohnB”)


Russell

• A logic of definite descriptions


Hilbert, Robinson

• A logic of indefinite descriptions


Russell, Hilbert, Robinson

“John wants a sloop.”

relational notional


The formal philosophy of Montague

• Relentless pursuit of compositionality“In the tradition of Montague grammar ... The interpretation of a language is given by a homomorphism between an algebra of syntactic representations and an algebra of semantic objects” http://en.wikipedia.org/wiki/Principle_of_compositionality

• “John wants a sloop.”

http://en.wikipedia.org/wiki/Principle_of_compositionality




Hall-Partee Summary of Montague

http://hiphilangsci.net/2013/05/01/on-the-history-of-the-question-of-whether-natural-language-is-illogical/


Hall-Partee Summary

“… the big difference is that Montague derives the compositionally; the semantic structure is homomorphic to the syntactic structure”

http://hiphilangsci.net/2013/05/01/on-the-history-of-the-question-of-whether-natural-language-is-illogical/


Semantic Parsing of Steedman

• Inducing semantic parsers from data, e.g, induce language model fragments from abstracted representations of entailment pairs.

“– Parsing directly coupled with compositional assembly of meaning representation or “logical form”;

– More recently, the induction of such parsers from data consisting of string meaning pairs.”

http://yoavartzi.com/sp14/slides/steedman.sp14.pdf


<pair id=“H23-4-O” correct_label=“N”><query> A privately appointed agent can be a minor, but acts of agency by a minor can be rescinded by the statutory agent. </query><article> (Agent's Capacity to Act)Article 102 An agent need not to be a person with the capacity to act. </article></pair>

Entailment pairs as data

• Queries that only one team answered correctly (NII)

19


Semantic Parsing of Steedman

http://yoavartzi.com/sp14/slides/steedman.sp14.pdf


Possible intermediate representations

http://people.cs.pitt.edu/~wiebe/courses/CS3730/Spring2013/cd.pdf


A spectrum of inference

Correct

Plausible/consistent/ambiguous

Interesting

Speculative

Wild-ass guess

Implausible

Wrong

deductive

Non-deductive

1 ,

2 ,

3 ,

n


Measures of entailment quality

• This strategy creates elaborate “semantic parsers” which embed (fragments) of general inference processes

• For explanation, what properties are preserved?• Soundness, completeness

• Are there classes of explanation processes• Probability, summary, hypothesis


John still wants a sloop

“John indicated that he had been considering a variety of sail craft, and his favourite was a sloop.”

“John wants a sloop.”?

1Sentential classifier with 83% accuracy measure

2Semantic parsing shows “looking at a sloop” Subsumes “wanting a sloop.”

3Model shows highest statistical correlation between these two sentences


Summary

• Entailment processes need to be developed to conform to inference expectations

• We need annotated corporate to build semantic parsing, summary relationships, IR models, etc. by ML techniques

• Quality of entailment, like accuracy of inference, should be assessed by “explainability”

Identifying the tradeoffs in textual entailment: deep representation versus shallow entailment Randy...

Documents

Transcript of Identifying the tradeoffs in textual entailment: deep representation versus shallow entailment Randy...