MASTERS THESIS DEFENSE MASTERS THESIS DEFENSE Solving Winograd Schema Challenge: Using Semantic...

1

MASTERS THESIS DEFENSE

Solving Winograd Schema Challenge: Using Semantic Parsing, Automatic Knowledge Acquisition

and Logical Reasoning

School of Computing, Informatics, and Decision Systems Engineering Arizona State University

BY

ARPIT SHARMAA DV I S O R: D R. C H I TTA B A RA L

O C T O B E R 3 1 S T 2 0 1 4

2

Presentation Overview Background and Motivation Problem and Related Work The System Semantic Parser & Pronoun Extractor Automatic Background Knowledge Extractor Logical Reasoning Engine

System Evaluation and Error Analysis Contributions and Future Works


Solving Winograd Schema Challenge: Using Semantic Parsing, Automatic Knowledge Acquisition and Logical Reasoning

3School of Computing, Informatics, and Decision Systems Engineering Arizona State University


Background and Motivation



Background One of the goals of AI : simulation of human-level intelligence in machines Ability to think and reason, based on the commonsense knowledge about things How to measure ? Turing Test in 1950 (Deceive humans in conversation) Not an ideal test

A conversation with Scott Joel Aaronson, computer scientist the MIT



Background

Hector J. Levesque suggested the Winograd Schema Challenge as an alternative to the Turing test in 2011

Its aim is not to deceive humans, but simulate human-like reasoning process

6

The town councilors refused to give the demonstrators a permit because they feared violence.

The town councilors refused to give the demonstrators a permit because they advocated violence

Contains a pair of sentences that differ in only one or two words The sentences contain an ambiguity that is resolved in opposite ways in the two sentences Requires the use of world knowledge and reasoning for its resolution



Winograd SchemaExample

7

A Question Answering test A Collection of 141 Winograd Schemas. 282 Total Sentences A Question about each Sentence.



The Winograd Schema Challenge

The town councilors refused to give the demonstrators a permit because they feared violence.

Who feared violence ?

The town councilors refused to give the demonstrators a permit because they advocated violence.

Who advocated violence?

Example

8

Helpful in: Text Summarization Reading Comprehension Deep Question Answering Ultimate Thinking Machines



Motivation



Problem and Related Work



The Problem

The Fish ate the worm because it was hungryWho was hungry ?

Of course Quagmire, the answer is “the fish”

Hey Peter, can you answer the above

question based on the sentence ?

How did you know ? The sentence does not

mention it.

Ooo!!!!Does that mean

I am GOD!!!!

No Peter!!! You are just a fat

HUMAN!!



The Problem

Humans have commonsense or background knowledge about things and events

How do humans get this knowledge ?

And, from where ?



Resolving Complex Cases of Definite Pronouns:The Winograd Schema Challenge

By Altaf Rahman and Vincent Ng, Human Language Technology Research Institute, 2012 Used statistical techniques and machine learning framework to combine their results (ranking-based approach) Created a new, Winograd Schema Challenge like, corpus.941 Winograd Schema (30% test set) 73% accuracyContains redundancy John shot Bill and he died.

The man shot his friend and he died.

Related Work



GoogleLions eat zebras because they are predators

Queries:“lions are predators”

“zebras are predators”

What if the sentence is, “Lions eat zebras because

they are hungry”


Related Work



Narrative Chains“partially ordered set of events centered around a common protagonist” - Nathanael Chambers, 2010

borrow-s invest-s spend-s pay-s raise-s lend-s

Drawbacks Only events (verbs) Less in number

The Fish ate the worm because it was hungryWho was hungry ?


Related Work



By Peter Schuller, Marmara University, Department of Computer Engineering, 2014 Converted the given sentence into a dependency graph Manually created background knowledge graph Combined both graphs to get the answer Shows usability on 4 Winograd Schema

Background Knowledge

Tackling Winograd Schemasby Formalizing Relevance Theory in Knowledge

Graphs

Related Work



The System



The WorkflowGiven Sentence and Question

Answer

Automatic Background Knowledge Extractor

Logical Reasoning Module

Background Sentence

Semantic Representation of the Sentence and question

Semantic Representation of the Background Sentence

Pronoun Extractor

Semantic Parser

Pronoun to Be Resolved



Semantic Parser & Pronoun Extractor



Semantic ParserRepresent text into an Expressive Formal Representation

PreserveGrammatical

Structure

Syntactic Dependency Parse

Distinguish words with same

conceptual sense

Ontology (WordNet)

Uses General Set of Relations

Knowledge Machine (KM) Slot

Dictionary



Semantic Parser

Stanford Dependency Parse of “The man loves his wife”

Syntactic Dependency Parse

lovesVBZ

manNN

wifeNN

hisPRP$

TheDT

det

nsubj dobj

poss



Semantic Parser

Semantic Parse of “The man loves his wife”

Knowledge Machine Slot Dictionary Mapping

lovesVBZ

manNN

wifeNN

hisPRP$

agent recipient

possesed_by

Mapping Stanford Dependency

Relations to KM Slot Dictionary Using Intuitive

Rules

22

superclass

instance_of instance_of



Semantic Parser

Ontology Addition to the Semantic Parse of “The man loves his wife because she loved him”

Ontology Addition

loves_3

man_2 wife_5

his_4

agent recipient

possesed_by

man

instance_of

instance_of

instance_of

loved_8

she_7 him_9

agentrecipient

caused_by

person

love


person

personemotion

superclass

his

wife

superclass



Pronoun Extractor

The man could not lift his son because he is so weak.

lift_5

man_2 son_7

his_6

agent recipient

possesed_by

man

instance_of

he_9

weak_12

trait

person

instance_of

instance_of

participantlift

not_4

negative

q_1

weak_3q

weak

trait

instance_of

instance_of

Who is weak?

weak

instance_of

he

son his

superclass

superclass

25

The Idea is, to learn the usage of English words and the contexts in which they are used




Creating query by using formal representation of the given sentence and the question

Extracting background knowledge sentences from a big source of raw text

That is done by,

26

Causal

Non Causal

Temporal

Locative




The fish ate the worm because it was tasty.

Mary took out her flute and played one of her favorite pieces. She has had it since she was a child.

Jackson was greatly influenced by Arnold, though he lived two centuries earlier.

Sam’s drawing was hung just above Tina’s and it did look much better with another one above it.

Categorization of Winograd Schema

27

Two subtypes of Causal category are solved by the system



Type2 : Causal AttributiveExample:-The man could not lift his son because he is so weak.

Who is weak?

Type1 : Direct Causal EventsExample:-Ann asked Mary what time the library closes, but she had forgotten.

Who had forgotten?




Automatic Background Knowledge ExtractorCreating Queries

The man could not lift his son because he was so weak .

Who was weak ?

Query Set 1 (Q1):

“.*not.*lift.*because.*weak.*”

“.*not.*lift.*because.*so.*weak.*”

Queries Type1: Use semantic graph of the given sentence and the question

Trace all nodes of the question into the given sentence (except “Wh” nodes)

Extract semantically important words (except entities)

Also consider the connective words

Combine the words in their order of occurrence in the sentence and join them using wildcard (.*) and quotes (“”)




lift_4

man_2 son_7

his_6

he_9 weak_12

not_3

so_11

q_1

weak_3

man son

not

weak

so

he

hisperson

weakagent

negative

traitrecipient

instance_of

agent


instance_of

instance_of

trait

instance_ofinstance_of possesed_by

superclass

superclass

superclass

Sentence Question

30

Queries Type2: Replace verbs with synonyms in query type 1.

Consider all combinations





Who was weak ?

A query among Q1 = “.*not.*lift.*because.*weak.*”

Query Set 2 (Q2):

“.*not.*pick.*because.*weak.*”

Final Queries: Final Set of Queries (Q) = Q1 Q2∪

Final Query Set (Q):

“.*not.*lift.*because.*weak.*”

“.*not.*lift.*because.*so.*weak.*”

“.*not.*pick.*because.*weak.*”

31

Using big source of raw text

Use search engine



Automatic Background Knowledge ExtractorExtracting Background Knowledge

Sentences

32

Two ways in which sentences are extracted from WWW

Example Query: “.*not.*lift.*because.*weak.*”




Sentences

33

Filtering the extracted sentences Should not contain the original sentence Should contain all the words in the query (in

any form) Should not contain partial sentences




Sentences


Query: “.*not.*lift.*because.*weak.*”

Filtered sentences: She could not lift it off the floor because she is a weak girl She could not even lift her head because she was so weak I could not even lift my leg to turn over because the muscles were

weak after surgery …..



Automatic Background Knowledge ExtractorParsing the Background Sentences

She could not lift it off the floor because she is a weak girl



Logical Reasoning Engine




Given Sentence


(ASP Rules)

Answer

Background Knowledge Sentence

Background Knowledge Sentences

Pronoun

37

Answer Set Programming Represent the Semantic Representation of the

Given Sentence and the Background sentence in ASP predicates Use ASP Reasoning Rules




38

Winograd SentenceAnn asked Mary what time the library closes, but she had forgotten

has(winograd,asked_2,agent,ann_1).has(winograd,asked_2,recipient,mary_3).has(winograd,asked_2,instance_of,ask).……..



Logical Reasoning EngineRepresenting the Winograd and the

Background SentencesBackground SentenceBut you asked me the security question but I forgotten

has(background,asked_103,agent,you_102).has(background,asked_103,instance_of,ask).has(background,ask,superclass,communication).……..

asked_2

ann_1 Mary_3ask

agent

instance_of

agent

asked_2

you_102

agent

ask

instance_of

communication

instance_of



Logical Reasoning EngineASP rules to capture general properties in Background and

Winograd sentences Reachability (Transitivity within context)

Cross context siblings (words belonging to same class in different contexts)

Negative Words (words with negative word associated with them)



Logical Reasoning EngineReachability

Background

reachableFrom(background, asked_3,forgotten_10)

41

Basic transitivity relationship between event nodes in a particular context.

reachableFrom(C,X,Y) :- has(C,X,REL,Y), context(C), eventRelation(REL).

reachableFrom(C,X,Z) :- reachableFrom(C,X,Y), has(C,Y,REL,Z), context(C) , eventRelation(REL), X!=Y, Y!=Z.



Logical Reasoning EngineReachability

Event Relations from KM

causes

caused_by

defeats

defeated_by

enables

enabled_by

inhibits

inhibited_by

……. (15 more)



Logical Reasoning EngineCross-Context Siblings

Winograd Background

crossContextSiblings(asked_2,asked_3)

43

Words in different sentences (Winograd or Background) are instances of the same conceptual class then they are defined as cross context siblings

crossContextSiblings(E1,E2) :- has(background,E1,instance_of,C),

has(winograd,E2,instance_of,C),

E1!=E2.



Logical Reasoning EngineCross-Context Siblings



Logical Reasoning EngineNegative Polarity

negativePolarity(lift_4)

45

Words associated with a negation word like ”not”, are defined by negativePolarity predicate.

negativePolarity(E) :- has(C,E,negative,N1), context(C).



Logical Reasoning EngineNegative Polarity



Logical Reasoning EngineType Specific Reasoning

Type1: Direct Causal Events

A B P

EVENT1 EVENT2

X Y X

EVENT1’ EVENT2’

rel1 rel1rel2rel3rel3 rel4

Winograd Background

Ann asked Mary what time the library closes, but she had forgotten.

Who had forgotten?



Logical Reasoning EngineType Specific Reasoning

Type1: Direct Causal Events

matchingEvents(asked_2,forgotten_13,asked_3,forgotten10)

Winograd Background



matchingEvents(A,B,A1,B1) :- crossContextSiblings(A,A1), reachableFrom(winograd,A,B),crossContextSiblings(B,B1) ,reachableFrom(background,A1,B1) , negativePolarity(A),not negativePolarity(B),negativePolarity(A1),not negativePolarity(B1).

Logical Reasoning EngineType1: Direct Causal Events

Step1: A and B are the reachable nodes in the sentence graph which has A1 and B1 as crossContextSibling, reachable events respectively from Background sentence graph.

49

Pronoun to be resolved



eventSubgraph(winograd,forgotten_13,agent,she_11)


Winograd

50

Step2: Extract the sub graph from the Winograd sentence which contains the pronoun to be resolved, the event in which it participates and their relation.

eventSubgraph(winograd,A,S,X) :- matchingEvents(A,B,C,D), has(winograd,A,S,X),

toBeResolved(X).






forgotten_110

entity1_109

agent

forget

instance_of

forget

instance_of

Background

eventSubgraph(background,forgotten_110,agent,entity1_109)


52

Step3: Extract the sub graph from Background sentence which contains a matching event of the event to which the pronoun to be resolved is related in the Winograd sentence.

eventsubgraph(background,A1,S,X1) :- eventSubgraph(winograd,A,S,X),

matchingEvents(A,B,A1,B1),

has(background,A1,S,X1).






forgotten_110

entity1_109

agent

forget

instance_of

entity1

instance_of

Background

asked_103

entity1_104

recipient

entity1

instance_of

next_event

…

…

eventPronounRelation(background,asked_103,recipient)


54

Step4: Extract the event and relation from the Background graph. It is helpful in getting the final answer.

eventPronounRelation(background,D,S1) :- matchingEvents(A,B,C,D),

eventSubgraph(background,C,S,X1),

has(background,D,S1,X2),

has(background,X1,instance_of,X),

has(background,X2,instance_of,X).







hasCoreferent(she_11,mary_3)

Winograd

56

Step5: Extract the co-referent of the pronoun to be resolved from the Winograd sentence graph.

hasCoreferent(P,X) :- eventPronounRelation(background,C,S),

matchingEvents(A,B,C,D),

has(winograd,A,S,X),

toBeResolved(P), P!=X.




57

The ASP implementation is similar to the Type1 implementation

Some more type specific rules are used along with the general rules



Logical Reasoning EngineType2: Causal Attributive



System Evaluation & Error Analysis



System Evaluation

Total 282 sentences in WSC Causal category has >200 Causal sub-categories, Type1 and Type2, combined have 100 sentences Results

Total Number of Sentences Evaluated

Answered Background Knowledge Not Found

Answered Correctly

Answered Incorrectly

Percentage Correct

100 80 20 70 10 87.5



Error Analysis

20 out of 100 not answeredSuitable background knowledge was not found

Mark ceded the presidency to John because he was less popular



Error Analysis

Bob paid for Charlie’s college education, he is very grateful

10 out of 80 incorrectly answered Deeper analysis of background knowledge is required

I paid the price for my stupidity. How grateful I am.Background Sentence:

Winograd Sentence:



Contributions and Future Works



Contributions

Implemented a system to solve the Winograd Schema Challenge by using Background Knowledge

Implemented an approach to automatically extract commonsense knowledge

Co-Implemented a new semantic representation system (available at www.kparser.org)



Future Works

Solving other WSC categories

Participate in NUANCE’s competition

Creating a commonsense Knowledge Base

Solve Reading Comprehension and other problems



THANK YOU!!!

67

THANK YOU!!!

MASTERS THESIS DEFENSE MASTERS THESIS DEFENSE Solving Winograd Schema Challenge: Using Semantic...

Documents

Transcript of MASTERS THESIS DEFENSE MASTERS THESIS DEFENSE Solving Winograd Schema Challenge: Using Semantic...