Dimensions of Scalability in Cognitive Models Research Team: Carnegie Mellon University - Psychology Department

Dr. Christian LebiereDr. David ReitterDr. Jerry VinokurovMichael FurlongJasmeet Ajmani

Overview

• Goal: Scaling up high-fidelity cognitive models by– Composing models– Abstracting models– Running large networks of models

• ACT-UP: a toolkit view of cognitive architectures– Same validated functionality, different form

• Lemonade game: Reusing and integrating models• Language learning: Scaling up to network cognition• The Geo-Game: Bringing it all together

– Platform for experimentation and integration

Dimensions of Scaling

ACT-R Cognitive Architecture

• Computational implementation of unified theory of cognition

• Commitment to task-invariant mechanisms

• Modular organization• Parallelism but strong

attentional limitations• Hybrid symbolic/

statistical processes

Issues with Cognitive Modeling

• High-fidelity cognitive models provide very accurate models of all observable dimensions of cognition (time, accuracy, gaze, neural) but

• They are computationally intensive as they simulate all cognitive processes in full detail

• They are labor intensive to specify all aspects of cognitive performance (knowledge, strategies)

• They are specialized to a given task in a way that makes them difficult to compose and reuse

• They usually focus on single-agent cognition

Scaling Up Cognitive Modeling

• Enable the implementation of more complex cognitive models in a more efficient manner

• Scale up the application of cognitive models to simulate learning and adaptation in communities (e.g., 1,000 models in parallel)

• Enable reuse and composition of cognitive models similar to software engineering view

• Facilitate integration of cognitive models with other modeling and simulation platforms

• Improve maintenance, update and validation

The Approach

• Difficulties: ACT-R is heavily constrained already, and models are difficult to develop, reuse and exchange

• Constraints: Architectural advances require further constraints, e.g. more representational constraints

• Scaling it up: Complex tasks, broad coverage of behavior, multi-agent cognition and predictive modeling may motivate further architectural changes

• Solution: produce models at a higher abstraction level• Retain and emphasize key cognitive mechanisms• Abstract purely mechanistic model aspects• Precisely specify model claims, underspecify/fit rest• Benefits of abstraction in efficiency, scalability, reuse

SymbolicSymbolic

SubsymbolicSubsymbolic(Learning / (Learning / Adaptation)Adaptation)

deterministicdeterministic

non-deterministicnon-deterministicexplains explains empirical empirical variancevariance

Cognitive Strategy

specify:specify:

deterministicdeterministic

non-deterministicnon-deterministicexplains explains empirical empirical variancevariance

underspecify:underspecify:

Underspecified Models

(Lisp Functions)

ACT-UP vs ACT-R 6

• Declarative memory: chunks as objects– Explicit context specification; all

activation computations• Procedural memory: productions as

functions– Explicit conflict set groups; utility

reinforcement learning• ACT-UP is synchronous with serial

execution– Parallelism in process of being

implemented• Perceptual-motor modules being planned

Validation

• Against canonical ACT-R tutorial models data

Efficiency

• Sentence production (syntactic priming) model– 30 productions in ACT-R, 720 lines of

code– 82 lines of code in ACT-UP (3 work-

days)– ACT-R 6: 14 sentences/second– ACT-UP: 380 sentences/second

Scalability

• Language evolution model– Simulates domain vocabulary emergence

(ICCM 2009, JCSR 1010)– 40 production rules in ACT-R– Complex execution paths: could not prototype– 8 participants interacting in communities

• In larger community networks:– 1000 agents– 84M interactions (about 1 min. sim.

Each)– 37 CPU hours

Related Work

• Douglass (2009; 2010) on large declarative memories– Implementation through Erlang threads– Focus on scalability

• Salvucci (2010) work on supermodels– Integrating and validating independent models– Focus on instruction interpretation for generality

• Stewart and West (2007) work on Python-ACT-R– Similar deconstructive view of architecture– Integration with neural constructs

Future Work

• Complete validation against canonical model set; currently in beta testing; full release planned for spring 2011

• Possible collaboration with AFRL Mesa on implementation of finite-state-based systems

• Potential use in other projects (Minds Eye, Robotics CTA)• Allow optional parallelism where needed and desired• Implement perceptual and motor modules• Potential implementation in other languages (C++, Java) to

facilitate code-level integration with common frameworks

Reitter, D., & Lebiere, C. (2010). Accountable Modeling in ACT-UP, a Scalable, Rapid-Prototyping ACT-R Implementation. In Proceedings of the 2010 International Conference on Cognitive Modeling. Philadelphia, PA.

Lebiere, C., & Reitter, D. (2010). ACT-UP: A Cognitive Modeling Toolkit for Composition, Reuse and Integration. In Proceedings of the 2010 MODSIM conference. Hampton, VA.

Lebiere, C., Stocco, A., Reitter, D., & Juvina, I. (2010). High-fidelity cognitive modeling to real-world applications. In Proceedings of the NATO Workshop on Human Modeling for Military Application, Amsterdam, NL, 2010.

Cognitive principles in cooperative and adversarial games:

Metacognition transfers via ACT-UP

Communities (Teamwork)

Dyads (Dialogue)

Individuals

Networks (Distributed Knowledge)

Complex Tasks, Broad-Coverage Models

Controlled Tasks, High-Fidelity Models

ACT-UP: Rapid prototyping/Reuse

• Dynamic Stocks & Flows ACT-UP model– Winning modeling competition entry– Model written in < 1 person-month– Free parameters (timing) estimated

from example data– Model generalized to novel conditions

• Reuse of Metacognitive Strategy in the Lemonade Stand Game (BRIMS 2010)

Kevin A. Gluck, Clayton T. Stanley, Jr. L. Richard Moore, David Reitter, and Marc Halbrügge. Exploration for understanding in model comparisons. Journal of Artificial General Intelligence (to appear), 2010.

David Reitter. Metacognition and multiple strategies in a cognitive model of online control. Journal of Artificial General Intelligence (to appear), 2010.

David Reitter, Ion Juvina, Andrea Stocco, and Christian Lebiere. Resistance is futile: Winning lemonade market share through metacognitive reasoning in a three-agent cooperative game. In Proceedings of the 19th Behavior Representation in Modeling & Simulation (BRIMS), Charleston, SC, 2010.

Multi-agent Games

• 2x2 games such as the Prisoner’s Dilemma– Evolution of cooperation vs. competition– Memory-based expectations (Lebiere et al, 2001)

• Adversarial games (Paper Rock Scissors, Baseball)– Zero-sum competition where predictability is

fatal– Sequence-based expectations (Lebiere et al, 1998;

2003)

• Lemonade game (3 players)– Simultaneous cooperation and competition– Predictability can be desirable for cooperation

The Lemonade Stand Game

• In each iteration, each of three players chooses a location 1..12

• Payoff is proportional to the distance to left and right neighbors.

• Hidden moves (blind choice)• 1 game: 100 iterations, then reset

(no state across games)

Zinkevich (2010, unpublished)

Basic Strategies

• Random (unpredictable): choose random loc.

• Sticky (predictable): choose same location– Roll, SquareRoot

• Tournament with those four agents– Equal performance

Strategy Elements

• Offer Cooperation: Be predictable• Predict: Learn patterns of opponents• Maximize Utility: Choose highest

expected payoff• Cooperate: Pick “friendly” opponent

whose payoff is also maximized• Monitoring: analyzing own/others

performance, keep history

Strategies

offer Coop Predict maxPayoff CoopMonitorSe

lfMonitorOp

p

Sticky +

StickySmart + +

StickySharp +

CopyCat + +

Statistician +

Cooperator + + +

Strategist + + +

Metacognition

• Facility to constantly monitor performance, and to adapt behavior accordingly

• Choose the best-performing strategy out of a set of strategies (Flavell 1979, Brown 1987)

• Strategy-shifting assumed in Dynamic Stocks & Flows data (DSF Challenge)

General Metacognition

• Prediction of each opponent’s next move– Learn from agent’s history in this game– Multiple possible representations and

pattern-matching• Action: Making a move

– Optimize Utility– Suggest cooperation– Cooperate– Hurt the worst adversaries

Evaluating Strategies

• Prediction and Action strategies are learned as episodes (instances):– Each prediction strategy per iteration, per

opponent– one action strategy per iteration

• Instance-based learning (Gonzales&Lebiere 2003)

• Objective: Prediction quality/Action payoff• Blending: weighted mean (recency,

frequency, objective as above)

Metacognition in Prediction

ACT-R Activation(recency, frequency)

Expected success of strategy s and

agent a

Episode in memory:time t, actual chosen location l of

agent a, predicted probability p for l,a by strategy s

• Each prediction strategy suggests a next location for each opponent

• All past predictions are stored throughout the game: <t,l,p> (time, actual location, predicted probability of that location)

Metacognition for Actions is similar

as in Reitter (2010) - DSF model

Evaluation

• Outcome of each strategy depends on configuration of players– Some strategies

will cooperate• Metacognitive

strategy is flexible, achieves consistently high results

• Bigger circle: higher winnings. Darker circle: consistent results.

Tournament

Adaptive Multi-Agent Behavior

• Offering cooperation and cooperating with the right opponent are crucial to doing well

• Metacognitive layer allows an agent to trump all others through generality and adaptivity

• Research questions:– Human performance in cooperative games:

issues of trust, social and cultural biases– Memory activation and rational retrieval

expectations as proxy for weighing past strategy success – limits of metacognition

Future Work

• ACT-R/ACT-UP’s learning vs. more basic Bayesian models: is cognitive learning more robust through open-endedness?

• Break down current limits of cognitive models generality– Are canonical architectural parameters optimal through

coevolution for empirical clustering factors and degrees?• Key part of environment is social interactions

– Automatic acquisition of rules, strategies, structural representations rather than modeler specification

– Metacognition: accumulation of micro-strategies library into reusable, general-purpose metacognitive layer

– Combination of above provide way of breaking out of task-specific models and their assumptions: beyond task-specific parameters, representation, strategies

Scaling Up Cognitive Models from Individuals to Large NetworksThe case of communication in human communities

Communities (Teamwork)

Dyads (Dialogue)

Individuals

Networks (Distributed Knowledge)

Complex Tasks, Broad-Coverage Models

Controlled Tasks, High-Fidelity Models

David Reitter and Christian Lebiere. Towards explaining the evolution of domain languages with cognitive simulation. Cognitive Systems Research (in press), 2010.

Dr. Christian LebiereDr. David ReitterCarnegie Mellon University

Interactive Alignment

• Garrod & Pickering 2004:

from

: G

arr

od

&P

ickeri

ng

, B

BS

20

04

Syntactic Representation

Syntactic Representation

Adaptation in Language

• Rapid decay within 8-10 secondsexperimentally, for selected constructions: Levelt & Kelter (1982),Branigan et al. (2000)

• Long-term adaptation effects, which do not decay, have also been observed (Comprehension: Mitchell et al. 1995. Production: Bock&Griffin 2000)

• ACT-R’s declarative memory decay explains the repetition probability decay

(Switchboard corpus) Reitter (2008)

Interactive Alignment

from

: P

ickeri

ng

&G

arr

od

, B

BS

20

04

Lexical Representation

Lexical Representation

Syntactic and Lexical Adaptation Predict Task

Success!(Reitter & Moore 2007)

Domain Language Experiment

• Vocabulary: Signs as meaning-signifier combinationSimple Communication System: Lewis 1989, Hurford 1989, Oliphant&Batali 1996

• Naming game: an idealized transaction between two players– Pictionary: a director draws a given target

concept using elementary drawings; a matcher has to guess the concept.

– 20 target concepts, repeated– Director/Matcher receive no explicit feedback“Brad Pitt”

Fay et al., Cognitive Science 34(3), 2010. Kirby et al., PNAS 2008; Fay et al. PhilTransRoySoc-B 2008

Pictionary Performance

Community IsolatedPairs

part

ner

swit

ch(c

om

mu

nit

ies)

part

ner

swit

ch(c

om

mu

nit

ies)

part

ner

swit

ch(c

om

mu

nit

ies)

From data by Fay et al. 2010

(empirical)

ID accuracy:proportion of signs retrieved

Broad Questions

• How does the architecture of human cognition interact with social structure?

• Have the human mind and large-scale social structures co-evolved?

• Can modeling predict the kinds of team structures that will yield optimal communication and collaboration?

Pictionary Model in ACT-UP

• Ontology shared betweendirector and matcher– abstract target concepts– concrete drawings– link weight distribution

acquired from Wall Street Journal collocations

• Director chooses three related drawings to convey a target concept

• Choice is conventionalized• Decision-making and memory

retention modeled with ACT-UP

Ontology

weig

hted

link

Pictionary and Networks

part

ner

swit

ch(c

om

mu

nit

ies)

part

ner

swit

ch(c

om

mu

nit

ies)

Reitter&LebiereJournal of Cognitive Systems Research, in press

(ACT-UP model)p

art

ner

swit

ch(c

om

mu

nit

ies)

ID accuracy:proportion of signs retrieved

100 rep.

Community IsolatedPairs

Scaling up to Networks

Communities (Teamwork)

Dyads (Dialogue)

Individuals

Networks (Distributed Knowledge)

Complex Tasks, Broad-Coverage Models

Controlled Tasks, High-Fidelity Models

Dr. Christian LebiereDr. David ReitterCarnegie Mellon University

Reitter, D., & Lebiere, C. (2010). Did social networks shape language evolution? A multi-agent cognitive simulation. In Proc. Cognitive Modeling and Computational Linguistics Workshop (CMCL 2010), Uppsala, Sweden.

Research Questions

• Does network structure affect convergence towards a common community vocabulary?– Or: Is declarative memory robust w.r.t. a variety of

network structures?• The small-scale, empirical and modeling data

suggest that extreme networks (fully vs. disconnected) arrive at similar performance, but converge differently. How? Why?

• Larger communities that differ in their connectivity are needed to answer these questions.

Network Types

• In a network, only network neighbors play the naming game• Social: Small-World network

(low path length, high clustering coefficient, assortatively mixed by degree)

• Grid (torus)• Random Graph• Organizational: Trees

• Controlled: mean degree (except trees), number of nodes

• Here: 512 nodes, mean deg. 6., 50 rep. per condition

ID Accuracy: Neighbors

Network type: ***random<grid<smallworld<tree

MCMC on LMER log(IDacc) ~ center(round) + cond + (1|sequence)

pre

lim

inary

resu

lts

ID Accuracy: Random Pairs

Tree

RandomSmall World Grid

Indication of convergence towards common vocabulary across network (measured after round 35)

Tree vs. others: n.s. (p=0.14, MCMC on LMER log(IDacc) ~ cond + (1|

sequence))pre

lim

inary

resu

lts

Summary

• Online Linguistic Adaptation is a known phenomenon – syntactic, lexical. Between two and more

participants.– Nodes can adapt to their immediate surroundings

• Tree hierarchies function very well when stable, but are not robust to structural change– Tree hierarchies represent contemporary

organizational hierarchies and generalize typical command structures

– Small-World structures are more robust to change.

Future Work

• Which advantages do non-tree network organizational structures have in situations where environment/ground truth changes, where adversarial elements are present?

• How can temporal dynamics in network structure (gradual ramp-up in connectivity) support information convergence (domain vocabulary acquisition)?

• Do cognitive models require explicit information processing policies in non-tree hierarchies, such that accountability and reliability are preserved?

• Integration of communication with planning, control and decision-making in complex dynamic domains.

Information Exchange in Networks

• Simulation at cognitive level: Language Evolution Model

• Simulation with Bayesian LearnersWang et al. (CMU Robotics), for a Bayesian Belief Update network

• Empirical validation is rare– Real-time communication networks are rarely

studied– Most empirical datasets contain asynchronously

produced communication, lacking control over exchanged information (e.g., Enron or Twitter corpora)

Human Networks: Empirical Experiments with the Geo Game

Communities (Teamwork)

Dyads (Dialogue)

Individuals

Networks (Distributed Knowledge)

Complex Tasks, Broad-Coverage Models

Controlled Tasks, High-Fidelity Models

Dr. Christian LebiereDr. David ReitterPsychology, Carnegie Mellon University

Dr. Katia SycaraAntonio JuarezDr. Paul ScerriDr. Robin GlintonRobotics Institute, Carnegie Mellon University

Dr. Michael LewisUniversity of Pittsburgh

MURI Team: The Geo Game

Task allocation among

humans/agents

Task allocation among

humans/agents

Probabilistic models of human decision-making in network

situations

Probabilistic models of human decision-making in network

situations

Decentralized control search and planning

Decentralized control search and planning

Information fusionInformation fusion

Network performance as a

function of topology

Network performance as a

function of topology

Communication, evolution, language

Communication, evolution, language

CornellCornell GMUGMU CMU Psychology

CMU PsychologyMITMIT PittPitt

Level 1,3Level 1,3 Level 2Level 2Level 1,2Level 1,2

Level 1-2.5

Level 1-2.5 Level 1-3Level 1-3 Level 1,3Level 1,3Level 1Level 1

Level 1,2Level 1,2 Level 1-2.5

Level 1-2.5 Level 3 Level 3 Level 2Level 2

Level 1-3

Level 1-3

Level 1,2Level 1,2 Level 1,3, 4Level 1,3, 4Level 1-3Level 1-3

Level 4Level 4 Level 2Level 2

Level 3Level 3 Level 2, 3Level 2, 3

Adaptive automationAdaptive automation Level 1,2Level 1,2 Level 1Level 1

Level 1,2 Level 2

CMURobotics

CMURobotics

Level 1Level 1Scaling of cognitiveperformance and

workload

Scaling of cognitiveperformance and

workload

Level 1,2

The Geo Game

• Information exchange in human networks– On-line (real-time) communication– Medium to large networks (15 to 1,000 nodes)– Defined information needed to execute given

task– Information is spread throughout the network

• Natural language as a means to exchange communication– Often task-specific, controlled language

(e.g. radio communication)• Trade-off: communication vs. task executionThe Geo Game platform is being developed by CMU Psychology and CMU

Robotics

Geo Game: Participant’s Task

Geo Game as Platform

• Subjects are organized in a graph– vertices define communication channels:

subjects can only communicate with network neighbors

– Currently: small-world network• The Geo Game is a platform

– Current game: foraging task– Other variants: trading agents, varied information

types (stochastic, graded, discrete, etc.), other networks (trees, adversarial networks)

– Server&web-browser based system; remotely deployable

Geo Game: Push vs. Pull

• Basic manipulation: push vs. pull of information– Relevant to practical domain and theoretical cognitive

issues• Push condition:

– Post all relevant information - items in cities; path efficiency

– Maximize information at cost of overloading attention/memory

• Pull condition:– Specify needs and only answer/forward relevant

information– Minimize overload at cost of opportunities

Push vs. Pull: Scalability

• If communication aids task success, does this effect scale?

One group of 15 participantsSeptember 2010

Geo Game: Time-to-Response

• Question-Answer Pairs: time (Q to A) is power-law distributed

cf. Barabasi (2010)

Publications

• Reitter, D., & Lebiere, C. (2009). A subsymbolic and visual model of spatial path planning. In Proceedings of the Behavior Representation in Modeling and Simulation (BRIMS 2009). Best paper award BRIMS 2009.

• Reitter, D., & Lebiere, C. (2009). Towards Explaining the Evolution of Domain Languages with Cognitive Simulation. In Proceedings of the 9th International Conference on Cognitive Modeling. Manchester, England.

• Reitter, D., Lebiere, C., Lewis, M., Wang, H., & Ma, Z. (2009). A Cognitive Model of Perceptual Path Planning in a Multi-Robot Control System. In Proceedings of the 2009 IEEE International Conference on Systems, Man, and Cybernetics. San Antonio, Texas.

• Reitter, D., Juvina, I., Stocco, A., & Lebiere, C. (2010). Resistance is Futile: Winning Lemonade Market Share through Metacognitive Reasoning in a Three-Agent Cooperative Game. In Proceedings of the Behavior Representation In Modeling and Simulations (BRIMS 2010) Conference. Charleston, SC

• Reitter, D., & Lebiere, C. (2010). Did social networks shape language evolution? a multi-agent cognitive simulation. In Proc. Cognitive Modeling and Computational Linguistics Workshop (CMCL 2010, at Association for Computational Linguistics: ACL 2010), Uppsala, Sweden.

• Reitter, D., & Lebiere, C. (2010). Accountable Modeling in ACT-UP, a Scalable, Rapid-Prototyping ACT-R Implementation. In Proceedings of the 2010 International Conference on Cognitive Modeling. Philadelphia, PA.

• Reitter, D., & Lebiere, C. (2010). On the influence of network structure on language evolution. In R. Sun, editor, Proc. Workshop on Cognitive Social Sciences: Grounding the Social Sciences in the Cognitive Sciences (at Cognitive Science: CogSci 2010), Portland, Oregon.

• Lebiere, C., & Reitter, D. (2010). ACT-UP: A Cognitive Modeling Toolkit for Composition, Reuse and Integration. In Proceedings of the 2010 MODSIM conference. Hampton, VA.

• Lebiere, C., Stocco, A., Reitter, D., & Juvina, I. (2010). High-fidelity cognitive modeling to real-world applications. In Proceedings of the NATO Workshop on Human Modeling for Military Application, Amsterdam, NL, 2010.

• Reitter, D. & Lebiere, C. (in press). Towards explaining the evolution of domain languages with cognitive simulation. Journal of Cognitive Systems Research.

• Reitter, D. & Lebiere, C. (in press). A cognitive model of spatial path planning. Journal of Computational and Mathematical Organization Theory.

• Reitter, D. (to appear). Metacognition and multiple strategies in a cognitive model of online control. Journal of Artificial General Intelligence.

• Gluck, K. A., Stanley, C. T., Moore, L. R., Reitter, D., & Halbrügge, M. (in press) . Exploration for understanding in model comparisons. Journal of Artificial General Intelligence.

Future Work

• The Geo Game can be exploited as an experimental platform for years to come– Communication: Alignment of communication

standards (e.g., vocabulary)– Knowledge: Information type and acquisition and its

influence on its distribution across the network (shared knowledge is not copied knowledge)

– Network structure: influence on team performance– Trust and strategy: Adversarial networks

• Mixed human/model/agent networks– Bootstrapping methodology for model validation– Information filtering for humans