Learning Plans from English Procedure...
Transcript of Learning Plans from English Procedure...
Machine Reading
- How can we get knowledge into computers?
- Machine reading: processing and understanding natural language-- accumulating relevant knowledge through inferences about text.
2
Approach 1: Read Children’s Stories
3
Jack was having a birthday party. Mother baked a cake.
Approach 1: Read Children’s Stories
- What is the relationship between Jack and Mother?
- Who will eat the cake?
- Where was Mother when she was baking the cake?
- Where was Jack?
- ... 3
Jack was having a birthday party. Mother baked a cake.
Eugene Charniak’s Ph.D. [1] “the chief concern motivating the model discussed here is relating a large body of knowledge to a particular story”
By preschool, children can represent and remember event sequences [2].
By the time they are reading, children have acquired a lot of world knowledge already!
4
[1] E. Charniak “Toward a Model of Children’s Story Comprehension.” 1972.[2] J. Wenner, P.J. Bauer. Bringing order to the arbitrary. 1999.
Approach 1: Read Children’s Stories
Big QuestionCharacterizing the problem:
How can we acquire knowledge by reading, when reading itself requires knowledge?
5
Big QuestionCharacterizing the problem:
How can we acquire knowledge by reading, when reading itself requires knowledge?
5
- Bootstrap solution: Gradually compose more new (more complex) representations out of existing (simpler) representations.
6
Approach 2: Use Commonsense KB
Jack was having a birthday party. Mother baked a cake.
is a gameis a children’s gameis a nickname for John.is a boy’s nameis a childhood game.is a lifting device.is kind of nickname.
(40)
Use OpenMind Common Sense knowledge base [1].
[1] http://commons.media.mit.edu
7
Approach 2: Use Commonsense KB
Jack was having a birthday party. Mother baked a cake.
is a gameis a children’s gameis a nickname for John.is a boy’s nameis a childhood game.is a lifting device.is kind of nickname.
(40)
bake a cake because you want toballoon used forlikely to find at toy balloonlikely to find at helium balloonbuying presents is forare fun
(17)
8
Approach 2: Use Commonsense KB
Jack was having a birthday party. Mother baked a cake.
is a gameis a children’s gameis a nickname for John.is a boy’s nameis a childhood game.is a lifting device.is kind of nickname.
(40)
bake a cake because you want toballoon used forlikely to find at toy balloonlikely to find at helium balloonbuying presents is forare fun
(17)
can care for a childis a womantake care of their childrenloves her childis part of my family
(260)
9
Approach 2: Use Commonsense KB
Jack was having a birthday party. Mother baked a cake.
is a gameis a children’s gameis a nickname for John.is a boy’s nameis a childhood game.is a lifting device.is kind of nickname.
(40)
bake a cake because you want toballoon used forlikely to find at toy balloonlikely to find at helium balloonbuying presents is forare fun
(17)
can care for a childis a womantake care of their childrenloves her childis part of my family
(260)
you should have an ovena birthday may make you want tobecause you want to celebrate a birthdayfirst thing add flour
(19)
10
Approach 2: Use Commonsense KB
Jack was having a birthday party. Mother baked a cake.
is a gameis a children’s gameis a nickname for John.is a boy’s nameis a childhood game.is a lifting device.is kind of nickname.
(40)
bake cake because you want balloon used forlikely to find at toy balloonlikely to find at helium balloonbuying presents is forare fun
(17)
can care for a childis a womantake care of their childrenloves her childis part of my family
(260)
Problem: Retrieving only the relevant knowledge
you should have an ovena birthday may make you want tobecause you want to celebrate a birthdayfirst thing add flour
(19)
11
Plans: a solution?
- are also called: plans, event chains, stories, scripts, narratives, procedures, task networks, sequential decision processes, ...
- Unite procedural and declarative semantic knowledge.
- Structure what is problematic, what questions are worth asking; embedding knowledge in a goal-driven problem solving context.
"Questions arise from a point of view–from something that helps to structure what is problematical, what is worth asking, and what constitutes an answer (or progress). It is not that the view determines reality, only what we accept from reality and how we structure it...”
-- Alan Newell in Artificial Intelligence and the Concept of Mind
12
Lexical-semantics of Events
- Verbs and their arguments reference lexical-semantic frames, or event structures [1].
- Properties of verbs (TimeBank)
- Tense: {none, present, past, future}
- Grammatical Aspect: {none, progressive, perfect, progressive perfect}
- Modality: {none, to, would, should, could, can, might}
- Polarity: {positive, negative}
- Event Type: { ?? }
[1] Levin B. and Hovav, Argument Realization 2005.
12
Lexical-semantics of Events
- Verbs and their arguments reference lexical-semantic frames, or event structures [1].
- Properties of verbs (TimeBank)
- Tense: {none, present, past, future}
- Grammatical Aspect: {none, progressive, perfect, progressive perfect}
- Modality: {none, to, would, should, could, can, might}
- Polarity: {positive, negative}
- Event Type: { ?? } - Need a semantic theory!
[1] Levin B. and Hovav, Argument Realization 2005.
13
Lexical-semantics of Events- Some compelling evidence of this relationship from
cognitive science:
- Verb aspect influences retrieval of event knowledge:
- imperfective form (was verb-ing) versus perfect form (had verb-ed) in a statement (e.g. “I was cooking/ I had cooked”) changes the retrieval of the location “kitchen” [1].
- When observing the same event, speakers of different languages attend to the event features relevant to organizing their languages specific verb lexicon, e.g. path versus manner [2].
- When reading about actions, people’s corresponding brain motor regions are activated [3].
[1] T. R. Ferretti, M. Kutas, and K. Mcrae. Verb aspect and the activation of event knowledge. 2007.[2] A. Papafragou, J. Hulbert, and J. Trueswell. Does language guide event perception? ... 2008. [3] R. A. Zwaan and L. J. Taylor. Seeing, acting, understanding: motor resonance in lang. comp. 2006.
14
Acquiring Plans the Open Mind Way
- Honda’s OMICS project [1], a clone of the Media Lab effort. Collecting knowledge exclusively about indoor common sense. High quality, manually reviewed.
- Has a parallel corpus of stories about ways to accomplish 174 different goals.
[1] http://openmind.hri-us.com/
15
access the internetact as a security guardanswer the doorbellanswer the phoneapply band aidassist person standing upassist someone in walkingboil the milkbuy from vending machinecall 911calm an infantchange a baby diaperchange a bulbchange bed sheetscharge a cell phonecheck for intruderscheck for weathercheck if a store is openchop vegetablesclean a spillclean the dishesclean the floorclean the showerclean the tableclean upclean up toysclose the blindsclose the curtainscook fishcook noodlecook pastacook ricedance with the childrendo laundrydraw the curtains
dry clothesdust an objectempty the kitchen sinkempty the trashentertain childrenerase the whiteboardfeed a childfeed a pet catfeed a pet dogfeed infantfeed the fishfetch a cold drinkfetch a ladderfetch an objectfill water in containerfind a personfind an objectfind out more informationfind the timefold clothesfollow someone aroundgather all scattered toysget food from refrigeratorget mailget the newspapergive a medicinegive a messagegive a message on phonego outsidegreet a visitorguard the househandle toxic materialshang clothesheat food in microwaveheat food on kitchen gas
help someone carry thingiron clotheskeep the dog awaykick a ballload the dishwasherlock up the houselock windowsmail a lettermake a bedmake a dinner reservationmake a flight reservationmake a listmake a presentationmake a shopping listmake a tossed saladmake baby sleepmake breakfastmake coffeemake fresh orange juicemake hot dogmake soupmake sure children fedmake teamake toasted breadmaking omelettemove furnituremow the lawnopen a web pageopen packageopen the garageopen the mailpack a mailing boxpack a suitcasepaint a wallpay bills
perform research on specphotocopy a paperpick up dishesplace ladder near wallplay a game on the compplay a movieplay a songplay pianoplug an electric appliance in plug battery into chargerpour beer into a glassprint documentpush someone in a wheelpush somethingput away groceriesput object awayput up a paintingraise the blindsread a story to a childrecharge batteriesremove and replace garbagereplace a refrigerator filterreplace a water tap filterreplace batteries in the treplace heater filterretrieve a toolsecure all exitssecure all windowssecure the perimeter of send a faxsend party invitationsserve a drinkserve a mealset a wake up alarmset the dining table
“sweep floor”
Problems with the corpus
“sweep floor”
P1. many ways to say the same thing!
Problems with the corpus
“sweep floor”
P2. Temporal Abstraction (nested events)
Problems with the corpus
“sweep floor”
P3. Global Alignment(problem of context)
Problems with the corpus
“sweep floor”
P4. Causal discontinuation (OR)
Problems with the corpus
1. Parse English sentences: clean, parse, extract predicate-argument structures.
2. Find global alignment of stories: Read in stories one at a time, align sequences, use structure to either:
a) detect missing nodes / context (alignment)
b) detect disjunctions (abstraction / is-a)
c) detect nested sequences (composition / part-of)
3. Infer corresponding state descriptions: Construct corresponding situation models for each step.
- Evaluation: the narrative cloze.
The Approach
22
Four example parsed narratives (of 36)
23
... and related background knowledge
1. Parse English sentences: clean, parse, extract predicate-argument structures.
2. Find global alignment of stories: Read in stories one at a time, align sequences, use structure to either:
a) detect missing nodes / context (alignment)
b) detect disjunctions (abstraction / is-a)
c) detect nested sequences (composition / part-of)
3. Infer corresponding state descriptions: Construct corresponding situation models for each step.
- Evaluation: the narrative cloze.
The Approach
25
Sequence Alignment- Sequential modeling work from NLP on language
models, and bioinformatics on modeling nucleic acids in DNA/amino acids in proteins.
- Needleman and Wunsch sequence alignment algorithm. Given two strings, A and B, populate a Score matrix and use back-pointers to retrieve the best alignment.
- Gap penalty:
- Beyond 2 strings?
- Time Complexity for n=2 seq:
!OpenCost! (len(gap)! 1)GapExtensionCost
max (|A|, |B|)n
425 ! 1014
26
n=1, pre-
27
n=1, post-
28
n=2, pre-
29
n=2, post-
30
n=3discarded
31
n=4, pre-
32
n=4, post-
33
n=5 pre
34
n=5 post
35
n=7 pre
36
n=7 post
37
Needleman-Wunsch Results
38
Beyond string matching- Match(Ai, Bj) = {1,0}
- But, how well do these match?
- get(mail) get(letter)
- open(box) open(mailbox)
- close(door) shut(door)
- return(home) go(inside)
39
Beyond string matching- Match(Ai, Bj) = {1,0}
- But, how well do these match?
- get(mail) get(letter)
- open(box) open(mailbox)
- close(door) shut(door)
- return(home) go(inside)
[1] Jiang, J and Conrath, D: Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy. 1997
- Jiang & Conrath generalization similarity metric [1]
40
A more general description
L. De Raedt and J. Ramon. Deriving distance metrics from generality relations. 2008.
41
n=4, pre-
42
n=4, post-
JC similaritytext similarity
43
n=5 pre
44
n=5 post
JC similaritytext similarity
45
Needleman-Wunsch (Relational) Results
46
What about inferring hidden state?
go to the mailbox
open the mailbox
extract letter
in(mailbox, mail) at(home, Dustin) sealed(letter) closed(mailbox)
in(mailbox, mail) at(mailbox, Dustin) sealed(letter) open(mailbox)
in(hand,mail) at(mailbox, Dustin) sealed(letter) open(mailbox)
- Learning a Left-Right Hidden Markov Model or Context Free Grammar (searching a model space, inducing a model w/ some MDL type constraint)
- Good for dealing with sequential, incomplete data. Not good for relational rich data. Hard to augment background knowledge.
- Relational Markov Models? Logical Hidden Markov Models (LoHMMs)?
47
Possible Solutions
Relational Markov Models
C. Anderson, P. Domingos, and D.S. Weld. Relational Markov Models and their Application to Adaptive Web Naviation. 2002.
A hidden Markov model
and a semantic abstraction
hierarchy (a taxonomy).
Relational Markov Models
C. Anderson, P. Domingos, and D.S. Weld. Relational Markov Models and their Application to Adaptive Web Naviation. 2002.
A graph rank-based heuristic
on the state transitions is used
to generalize arguments,
in favor of a compact representation.
Problems Representational Solutions
P1. Lexical Ambiguity
P2. Temporal Abstraction
P3. Global Alignment
P4. Causal discontinuation
1. Taxonomic (Is-A) hierarchy
2. Compositional (Part-of) hierarchy
3. Sequential
An ideal representation?
Representational Abstraction
51
h3 = [shape(circle) ∧ taste(sweet) ∧ color(green)]
h0
h2h1
h5h4h3
generalization lattice
<- more general m
ore specific ->
lattice = a set, a partial ordering, and greatest upper/lower bound operations.
h0 = [shape() ∧ taste() ∧ color()]
A hypothesis space of abstractions (removing details).
Representational Composition- Bootstrap solution: Gradually compose more
new (more complex) representations out of existing (simpler) representations.
52
composition
appletaste(sweet)
color(light-green)
shape(circle) composition
Structural versus Assertional
Various encodings of the statement “John loves tacos:”
LOVES: SUBJECT: John OBJECT: tacos
Loves(John,tacos)
tacos
loves
John
Different structures, same assertions!
Structural knowledge is the internal organization of the knowledge while assertional knowledge is the set of claims it makes about the world.
Relational Sequence Alignment
[1] Much related work by Kristian Kersting and Luc De Raedt.
- Relational (in first order logic), instead of propositional, descriptions of states.
Predicates/arity: vi/2, cd/1, ls/0, pdfview/2
Ground atoms -- predicates with non-variable terms:
vi(ch2,tex)
Ground clauses:
Generalized Clauses (with variables):
cd(X), vi(Y, tex), latex(Y, tex)
Sequential
Relational Sequence Alignment
[1] Much related work by Kristian Kersting and Luc De Raedt.