1 Episodic Memory for Soar Agents Andrew Nuxoll 11 June 2004.
-
date post
20-Dec-2015 -
Category
Documents
-
view
215 -
download
1
Transcript of 1 Episodic Memory for Soar Agents Andrew Nuxoll 11 June 2004.
1
Episodic Memory for Soar AgentsEpisodic Memory for Soar Agents
Andrew NuxollAndrew Nuxoll11 June 200411 June 2004
2
What is Episodic Memory?What is Episodic Memory?
Memories of specific events in our pastMemories of specific events in our past Example: Your last vacationExample: Your last vacation
3
WorkingMemory
Soar-EpMem in ActionSoar-EpMem in Action
Cue
Retrieved
Episodes
A simple example…A simple example…
4
WorkingMemory
Soar-EpMem in ActionSoar-EpMem in Action
Cue
Retrieved
Episodes
New memories are recorded periodicallyNew memories are recorded periodically
5
WorkingMemory
Soar-EpMem in ActionSoar-EpMem in Action
Cue
Retrieved
Episodes
The agent creates a cueThe agent creates a cue
6
WorkingMemory
Soar-EpMem in ActionSoar-EpMem in Action
The cue is matched to episodic memoryThe cue is matched to episodic memory
Cue
Retrieved
Episodes
7
WorkingMemory
Soar-EpMem in ActionSoar-EpMem in Action
Cue
Retrieved
Episodes
The best match is retrieved into WMThe best match is retrieved into WM
8
Benefits of Episodic MemoryBenefits of Episodic Memory
Aids in decision making through predicting the Aids in decision making through predicting the outcome of possible courses of actionoutcome of possible courses of action
A recorded history can be used to answer A recorded history can be used to answer questions about the pastquestions about the past
To help keep track of progress on long-term To help keep track of progress on long-term goalsgoals
Learn from past experience when new Learn from past experience when new time/resources become availabletime/resources become available
Generalize knowledge by comparing multiple Generalize knowledge by comparing multiple events simultaneouslyevents simultaneously
9
Previous WorkPrevious Work
Episodic MemoryEpisodic Memory Psychology – Endel TulvingPsychology – Endel Tulving Cognitive Modeling – Erik AltmannCognitive Modeling – Erik Altmann
Case-Based ReasoningCase-Based Reasoning Continuous CBR – Ram and SantamarContinuous CBR – Ram and Santamarííaa
10
Key Differences:Key Differences:
Episodic Memory ResearchEpisodic Memory Research Architectural implementationArchitectural implementation Domain independentDomain independent
Continuous Case-Based ReasoningContinuous Case-Based Reasoning Qualitative vs. quantitative episode Qualitative vs. quantitative episode
contentcontent Differing scope of match and retrievalDiffering scope of match and retrieval
11
Pilot Implementation: EncodingPilot Implementation: Encoding
Encoding initiation:Encoding initiation: upon significant upon significant change in activation levelschange in activation levels
Episode determination:Episode determination: hand selected and hand selected and domain specificdomain specific
Feature selection:Feature selection: hand selected and hand selected and domain specificdomain specific
12
Pilot Implementation: StoragePilot Implementation: Storage
Episode structure:Episode structure: episodes are stored as episodes are stored as Soar productionsSoar productions
Episode dynamics:Episode dynamics: none none
13
Pilot Implementation: RetrievalPilot Implementation: Retrieval
Retrieval initiation:Retrieval initiation: deliberate retrieval in an deliberate retrieval in an agent-selected substate agent-selected substate
Cue determination:Cue determination: agent-selected data agent-selected data Retrieval:Retrieval: exact match exact match Retrieved episode representation:Retrieved episode representation: direct direct
modification of the agent-selected substatemodification of the agent-selected substate Retrieval meta-data:Retrieval meta-data: unique sequential id (to unique sequential id (to
provide an idea of temporal order)provide an idea of temporal order)
14
Pilot Implementation: IssuesPilot Implementation: Issues
Exact match led to encoding specificity Exact match led to encoding specificity issuesissues
Problems from overwriting the sub-stateProblems from overwriting the sub-state RecursionRecursion Spurious operator proposalsSpurious operator proposals Requires that agent create a sub-state to Requires that agent create a sub-state to
do a retrievaldo a retrieval Domain dependentDomain dependent
15
Current Implementation: ChangesCurrent Implementation: Changes
Partial match over a separate episodic Partial match over a separate episodic memorymemory Memories are no longer stored as rulesMemories are no longer stored as rules
Use of an architecture-specified buffer for Use of an architecture-specified buffer for query and retrieval (analogous to the ^io query and retrieval (analogous to the ^io link)link)
16
Current Implementation: EncodingCurrent Implementation: Encoding
Encoding initiation:Encoding initiation: one episode per agent one episode per agent actionaction
Episode determination:Episode determination: all of working all of working memory(!)memory(!)
Feature selection:Feature selection: the entire episode can the entire episode can affect retrievalaffect retrieval
17
Current Implementation: StorageCurrent Implementation: Storage
Episode structure:Episode structure: episodes are stored in episodes are stored in an internal data structurean internal data structure
Episode dynamics:Episode dynamics: still none still none
18
Current Implementation: RetrievalCurrent Implementation: Retrieval
Retrieval initiation:Retrieval initiation: cue is constructed in an cue is constructed in an architecture-specified bufferarchitecture-specified buffer
Cue determination:Cue determination: agent selected data agent selected data Retrieval:Retrieval: exact match exact match Retrieved episode representation:Retrieved episode representation: the episode is the episode is
recreated in an architecture-specified bufferrecreated in an architecture-specified buffer Retrieval meta-data:Retrieval meta-data: agent can retrieve the next agent can retrieve the next
memory in temporal sequence memory in temporal sequence
19
Working Memory ActivationWorking Memory Activation
Extension of the memory decay work by Extension of the memory decay work by Ron ChongRon Chong
Reimplementation by Michael James:Reimplementation by Michael James: Includes all of working memory Includes all of working memory Improvements in efficiencyImprovements in efficiency
20
Activation & MatchingActivation & Matching
Problem:Problem: All WMEs in an episode are All WMEs in an episode are weighted equallyweighted equally
Core Idea:Core Idea: The activation level of WMEs The activation level of WMEs indicates their relevance to current taskindicates their relevance to current task
Implementation:Implementation: Use the activation levels Use the activation levels of the WMEs in the episode to bias the of the WMEs in the episode to bias the matchmatch
21
Evaluation using EatersEvaluation using Eaters
Pac-Man-likePac-Man-like Two types of foodTwo types of food
Bonus food (10 pts)Bonus food (10 pts) Normal food (5 pts)Normal food (5 pts)
22
Create a memory cue (input-link + proposed direction)Create a memory cue (input-link + proposed direction)
East
South
North
Evaluate moving in each available directionEvaluate moving in each available direction
An Episodic Memory EaterAn Episodic Memory Eater
EpisodicRetrieval
Retrieve the best matching memoryRetrieve the best matching memory
RetrieveNext Memory
Retrieve the Retrieve the nextnext memory (in temporal order) memory (in temporal order)Use the change in score to evaluate the proposed actionUse the change in score to evaluate the proposed action
Move North = 10 points
23
Initial ResultsInitial Results
24
Problem #1: I-Support MaskingProblem #1: I-Support Masking
Problem:Problem: Testing an Testing an i-supported WME i-supported WME provides no provides no activation boostactivation boost
Solution = Pay it Solution = Pay it Backward:Backward: Testing Testing an i-supported an i-supported WMEs boosts the WMEs boosts the activation level of its activation level of its “set of o-support”“set of o-support”
25
Problem #2: New WME MaskingProblem #2: New WME Masking
Problem:Problem: A new WME starts at a fixed A new WME starts at a fixed activation levelactivation level
Solution = Pay it Forward:Solution = Pay it Forward: Activation of Activation of newly created WMEs is based upon those newly created WMEs is based upon those WMEs which were tested to create itWMEs which were tested to create it
26
ResultsResults
27
Current Challenge: PerformanceCurrent Challenge: Performance
28
NuggetsNuggets CoalCoal
Domain independent, Domain independent, architectural architectural implementation implementation
Performance issuesPerformance issues Still only tested in a Still only tested in a
single domainsingle domain