NTL – Converging Constraints
description
Transcript of NTL – Converging Constraints
![Page 1: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/1.jpg)
NTL – Converging Constraints
• Basic concepts and words derive their meaning from embodied experience.
• Abstract and theoretical concepts derive their meaning from metaphorical maps to more basic embodied concepts.
• Structured Connectionist Models can capture both of these processes nicely.
• Grammar extends this by Constructions: pairings of form with embodied meaning.
![Page 2: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/2.jpg)
Simulation-based language understanding“Harry walked to the cafe.”
Schema Trajector Goalwalk Harry cafe
Analysis Process
Simulation Specification
Utterance
SimulationCafe
Constructions
General Knowledge
Belief State
![Page 3: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/3.jpg)
19Konvens, 09.10.2000
The ICSI/BerkeleyNeural Theory of Language Project
![Page 4: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/4.jpg)
Background: Primate Motor Control• Relevant requirements (Stromberg, Latash, Kandel, Arbib, Jeannerod,
Rizzolatti)– Should model coordinated, distributed, parameterized control programs
required for motor action and perception.– Should be an active structure.– Should be able to model concurrent actions and interrupts.
• Model– The NTL project has developed a computational model based on that
satisfies these requirements (x- schemas).– Details, papers, etc. can be obtained on the web at
http://www.icsi.berkeley.edu/NTL
![Page 5: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/5.jpg)
Active representations• Representation based on stochastic Petri nets captures dynamic,
parameterized nature of actions• Many inferences about actions derive from what we know about
executing them• Generative model: action, planning, recognition, language.
Walking:bound to a specific walker with a
direction or goalconsumes resources (e.g., energy)may have termination condition
(e.g., walker at goal) ongoing, iterative action
walker=Harry
goal=home
energy
walker at goal
![Page 6: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/6.jpg)
Active Motion Model
Evolving Responses of Competing Models over Time.
Nigel Goddard
1989
![Page 7: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/7.jpg)
Language Development in Children
• 0-3 mo: prefers sounds in native language• 3-6 mo: imitation of vowel sounds only• 6-8 mo: babbling in consonant-vowel segments• 8-10 mo: word comprehension, starts to lose sensitivity to
consonants outside native language• 12-13 mo: word production (naming)• 16-20 mo: word combinations, relational words (verbs, adj.)• 24-36 mo: grammaticization, inflectional morphology• 3 years – adulthood: vocab. growth, sentence-level grammar
for discourse purposes
![Page 8: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/8.jpg)
Learning Spatial Relation Words Terry Regier
A model of children learning spatial relations.Assumes child hears one word label of scene.Program learns well enough to label novel scenes
correctly.Extended to simple motion scenarios, like INTO.System works across languages.Mechanisms are neurally plausible.
![Page 9: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/9.jpg)
Learning System
We’ll look at the details next lecture
dynamic relations(e.g. into)
structured connectionistnetwork (based on visual system)
![Page 10: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/10.jpg)
Limitations
• Scale• Uniqueness/Plausibility• Grammar• Abstract Concepts• Inference• Representation• Biological Realism
![Page 11: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/11.jpg)
physics lowest energy state
chemistry molecularminima
biology fitness, MEU neuroeconomics
vision threats, friends
language errors, NTL
Constrained Best Fit in Nature
inanimate animate
![Page 12: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/12.jpg)
Learning Verb MeaningsDavid Bailey
A model of children learning their first verbs.Assumes parent labels child’s actions.Child knows parameters of action, associates with wordProgram learns well enough to: 1) Label novel actions correctly 2) Obey commands using new words (simulation)System works across languagesMechanisms are neurally plausible.
![Page 13: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/13.jpg)
![Page 14: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/14.jpg)
![Page 15: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/15.jpg)
Motor Control (X-schema) for SLIDE
![Page 16: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/16.jpg)
![Page 17: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/17.jpg)
Parameters for the SLIDE X-schema
![Page 18: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/18.jpg)
Feature Structures for PUSH
![Page 19: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/19.jpg)
![Page 20: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/20.jpg)
System Overview
![Page 21: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/21.jpg)
Learning Two Senses of PUSH
Model merging based on Bayesian MDL
![Page 22: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/22.jpg)
Training ResultsDavid Bailey
English• 165 Training Examples, 18 verbs• Learns optimal number of word senses (21)• 32 Test examples : 78% recognition, 81% action• All mistakes were close lift ~ yank, etc.• Learned some particle CXN,e.g., pull up
Farsi • With identical settings, learned senses not in English
![Page 23: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/23.jpg)
![Page 24: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/24.jpg)
![Page 25: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/25.jpg)
physics lowest energy state
chemistry molecularminima
biology fitness, MEU neuroeconomics
vision threats, friends
language errors, NTL
Constrained Best Fit in Nature
inanimate animate
![Page 26: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/26.jpg)
![Page 27: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/27.jpg)
Compositionality
• Traditional Context-free composition of logical forms
• Contemporary Constructional composition of conceptual frames
Formal Cognitive Linguistics
![Page 28: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/28.jpg)
Embodied Construction Grammar
(Bergen, Chang & Paskin 2000)• Assumptions from Construction Grammar
– Constructions are form-meaning pairs(Lakoff 1987, Goldberg 1995)
– Constructions vary in degree of specificity and level of description (morphological, lexical, phrasal, clausal)
• Constructions evoke and bind semantic schemas• Additional influences
– Cognitive Grammar (Langacker 1987)– Frame Semantics (Fillmore 1982)– Structured Connectionism (Feldman 1987)
![Page 29: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/29.jpg)
Phonetics
Semantics
Pragmatics
Morphology
Syntax
Traditional Levels of Analysis
![Page 30: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/30.jpg)
Phonetics
Semantics
Pragmatics
Morphology
Syntax
“Harry walked into the cafe.”
Utterance
![Page 31: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/31.jpg)
Language understanding: analysis & simulation
“Harry walked into the cafe.”
Analysis Process
Simulation Specification
Utterance
Constructions
General Knowledge
Belief State
construction WALKED in context cconstituents:
form f of type [wakt]meaning walking construed as Walk-Action
semantic constraints:walking.time before c.speech-timewalking.aspect = encapsulated
designates walking
CAFE Simulation
![Page 32: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/32.jpg)
Simulation specification
A simulation specification consists of:•semantic schemas evoked by constructions•bindings between schemas (labeled by the constructions that enforce them)
![Page 33: NTL – Converging Constraints](https://reader036.fdocuments.us/reader036/viewer/2022062222/56815d9c550346895dcbc29c/html5/thumbnails/33.jpg)
Conclusion• Language acquisition and use is a hallmark of being human
– Language seems to rely on fine-grained aspects of embodied (sensory-motor and social cognition) primitives and brain-like computation (massively parallel, distributed, spreading activation, temporal binding).
– Understanding requires imaginative simulation!– We have built a pilot system that demonstrates the use of motor control
representations in grounding the language of abstract actions and policies.• Sensory-Motor imagination and simulation is crucial in interpretation!
• Ongoing Work.– Formalize and use a compositional set of embodied conceptual primitives and
grammatical constructions.– Perform both behavioral and fMRI imaging experiments to test the predictions
of the simulation hypothesis. – Further refine and ground the model in details of neural anatomy and functional
architecture (basal-thalamic-cortical loops).