Rensselaer CognitiveScience
The Shape of Things to Come:An Emerging Constellation of Interconnected Tools for Developing the Right Cognitive Model at the Right ScaleWayne D. GrayProfessor of Cognitive ScienceProfessor of Computer Science
Opening Plenary Address for the 19th Conference on Behavior Representation in Modeling and Simulation (BRiMS 2010)
Charleston, South Carolina2010-Mar-23
Tuesday, April 6, 2010
GrayMatterTypewritten TextGray, W. D. (2010). The Shape of Things to Come: An Emerging Constellation of Interconnected Tools for Developing the Right Cognitive Model at the Right Scale. Paper presented at the Keynote Addressed delivered to the Behavior Representation in Modeling and Simulation Conference (BRiMS 2010), Charleston, SC.
Rensselaer CognitiveScience
Outline
• Discussion of problems with existing technologies for computational cognitive modeling
• Introduction of three very different tools
• Discussion of their interconnectedness
• The shape of things to come
2
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Major Problems with the Current Tools for Cognitive Modeling
• Ease of Use
• Toolbox
• Time Scale
• Generative Models
• Emergence
3
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Major Problems: Ease of Use
• Prerequisite Knowledge
• Computer Science
• Cognitive Science
• Domain Expertise
• Building Models
• 80/20 rule
• In large models of complex task, writing 80% of the code feels as tedious as writing in assembly language
• But, cannot get to the interesting 20% unless the other 80% is written
• Assessment: Modeling is too hard and takes too long
4
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Major Problems: Toolbox
• An ideal toolbox would have a perfect tool for each task
• Too many modelers, too much of the time, act as if the only tool in their toolbox was a hammer
• Assessment: To a man with a hammer, everything looks like a nail - there are a lot of hammers out there!
5
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Major Problem: Time Scale
Scale (sec) Time Units System Analysis World (theory)
1000000000 decades Technology Culture
Social & Organizational
100000000 years System Development Social & Organizational10000000 months Design
Education
Social & Organizational
1000000 week TaskEducation
Social & Organizational
100000 days
Task Traditional Task AnalysisBounded Rationality
10000 hours Task Traditional Task AnalysisBounded Rationality
1000 10 min
Task Traditional Task AnalysisBounded Rationality
100 min Subtask Strategies & Procedures
Bounded Rationality
10 10 sec Unit task Procedures & Methods
Cognitive Band (symbolic)1 1 sec
Interactive Routines
Embodiment Level (⅓ to 3 s) Cognitive Band (symbolic)0.1 100 ms Production System Elements (DME-MA-VA)
Cognitive Band (symbolic)
0.01 10 ms Atomic Components Architectural
Biological Band (subsymbolic)0.001 1 ms Parameters
Architectural Biological Band (subsymbolic)
6
Newell’s Analysis of Levels: The Time Scale of Human Action
Newell, A., & Card, S. K. (1985). The prospects for psychological science in human-computer interaction. Human-Computer Interaction, 1(3), 209–242.
Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Major Problems: Time Scale
• Human behavior transcends multiple time scales of human activity
• To understand complex behavior it may be productive to model at multiple time scales
• Example: There are many models of eye movements made during reading. To a nonmodeler interested in understanding reading comprehension, it might seem reasonable to scale up such a model to predict reading comprehension problems among children
• Assessment: Little reuse of models built to capture one time scale to modeling phenomena that are emergent at other time scales
• Time scale intolerance and a concomitant lack of time scale interconnectedness
7
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Major Problems: Generative & Predictive
• Static and Descriptive
• Too many models simply trace the cognitive, perceptual, and motor activities required to account for one particular instance of one task
• Generative and Predictive
• We want models that generate predictions in dynamic task environments where the exact order of events is not determined and the detailed features of the task environment may not be fixed
• Assessment: Limited generativity. Small changes to the task or task environment often require editing or major rewrites of model
8
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Major Problems: Emergence
• We would like to start our models with the basic facts or strategies of the novice performer and predict changes overtime with practice and experience
• Speed up performance
• Optimization of strategy selection
• Emergence of new strategies
• Emergence of new declarative knowledge
9
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Major Problems: Emergence
• Assessment:
• Cognitive modeling approaches (e.g., architectures, reinforcement learning, Bayesian modeling) do an adequate job of speed up and optimization of a fixed set of knowledge and strategies
• Almost no discovery of new strategies with practice
• Limited emergence of declarative knowledge - new facts, but not new types/categories
10
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Major Problems with the Current Tools for Cognitive Modeling
• Why are these issues important?
• An increasing recognition that human modeling of all types is important to our society and to national defense by enabling
• The design of human-machine and human-information systems tailored to the capabilities of the human
• Predictions of the reactions of populations to events
• Network Science
• Better training
• Tailoring system to different populations of users (e.g., aging)
11
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Outline
• Discussion of problems with existing technologies for computational cognitive modeling
• Introduction of three very different tools
• Discussion of their interconnectedness
• The shape of things to come
12
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Three Tools: LARGE CAVEAT
• The title of the talk is The Shape of Things to Come
• No assertion that
• The tools I mention are THE TOOLS OF THE FUTURE
• Rather, the intended claim is much more modest; namely,
• The level of ease, flexibility, and interconnectedness that these tools aspire to are what we must work towards building
13
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Descriptions and Report Cards for Three Tools
• ACT-R
• CogTool
• SANLab
14
Tuesday, April 6, 2010
Rensselaer CognitiveScience
ACT-R
• A community of approximately 200 modelers at 91 institutions worldwide
15
Tuesday, April 6, 2010
Anderson et al. (2004)
External World
Visual Module(Occipital/etc)
Manual Module(Motor/Cerebellum)
Manual Buffer(Motor)
Visual Buffer(Parietal)
(Execution (Thalamus)
Selection (Pallidum)
Matching (Stratium)
Retrieval Buffer(VLPFC)
Goal Buffer(DLPFC)
Intentional Module(not identified)
Declarative Module(Temporal/Hippocampus)
Prod
uctio
ns(B
asal
Gan
glia
)
Tuesday, April 6, 2010
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: ACT-R
• Ease of Use
• Prerequisite Knowledge: D
• Best to have at least a minor in Computer Science
• Masters or better in Cognitive Science
• & Domain expertise
Building Models: D+
• About as easy as using any programming language to build any large software system. In recent years specialized tools have been developed to assist in writing, debugging, and editing models
18
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: ACT-R
• Toolbox: C
• Limited Toolbox
• Many “closed form” ACT-R models written in spreadsheets to focus on single issues (e.g., memory, learning) that can be resolved by parameter fitting
• Full models that actually perform the task using the same software as people use
• SAL - Synthesis of ACT-R & Leabra
• Possible but required resources of two extremely strong laboratories (Anderson/Lebiere at CMU and O’Reilly’s at CU)
19
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: ACT-R
• Time Scales: C
• Carnegie Tutors are written in a pre-ACT-R language that (among other things) incorporate much larger temporal steps than ACT-R (3 s versus 50 ms)
• Is possible to write different models to focus on different levels of analysis, but this requires much advanced planning and is typically not done
20
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: ACT-R
• Generative: B+
• ACT-R models can adapt to a range of changes in the task environment if the strategies built into the models can be applied
• Emergence:
• Speed up with practice: A
• Optimization of strategy selection: A
• Emergence of new strategies: F
• Acquisition of new declarative knowledge: C
• (New facts, not new types)
21
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Descriptions and Score Cards for Three Tools
• ACT-R
• CogTool
• SANLab
22
Tuesday, April 6, 2010
Rensselaer CognitiveScience
CogTool: Build a Model without Knowing Anything about Modeling
• Developed by Bonnie John (CMU)
• http://cogtool.hcii.cs.cmu.edu/
• Target audience is HCI designers
• Storyboarding as modeling
23
Tuesday, April 6, 2010
http://cogtool.hcii.cs.cmu.eduhttp://cogtool.hcii.cs.cmu.edu
Rensselaer CognitiveScience
Create Storyboards
24
• Capture screenshots from existing applications or create your own storyboards (e.g., PhotoShop™, etc)
• Example: Modeling the time that an experienced user would require to add a recurrent event to Google Calendar using one of the several methods provided by Google
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Add hot spots
25
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Collect a Bunch of Screen Shots, Add Hot Spots, and Link
26
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Run Model
27
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Run Model
27
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Run Model
27
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Run Model
27
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Run Model
27
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Run Model
27
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Run Model
27
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Run Model
27
• Predicted expert performance time using this method to add a recurrent calendar event is: 20.342 s
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: CogTool
• Ease of Use
• Prerequisite Knowledge: A+
• Ability to read a manual and experience with modern graphic interfaces
• Domain Expertise
• Building Models: A+
• Very easy: More like storyboarding than modeling
28
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: CogTool
• Hammer versus Toolbox
• A hammer not a toolbox: F
• Keystroke Level GOMS - Predictions of Expert Performance Time
29
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: CogTool
• Time Scale: F
• Ability to go up and down a level of analysis easily
• One Level
30
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: CogTool
• Generative: D
• Minor edits are easy, but no true generative capability
• If environment changes model must be edited to reflect those changes
• Emergence: F
• No speed up in performance, no optimization of strategy selection, no new strategies, no new declarative knowledge
31
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Descriptions and Score Cards for Three Tools
• ACT-R
• CogTool
• SANLab
32
Tuesday, April 6, 2010
Rensselaer CognitiveScience
SANLab
• The Stochastic Analytic Network Laboratory (SANLab) for Cognitive Modeling
• Developed by Wayne Gray & Evan Patton (RPI)
• http://cwl-projects.cogsci.rpi.edu/sanlab/wiki/SANLab-CM
33
Tuesday, April 6, 2010
http://cwl-projects.cogsci.rpi.edu/sanlab/wiki/SANLab-CMhttp://cwl-projects.cogsci.rpi.edu/sanlab/wiki/SANLab-CMhttp://cwl-projects.cogsci.rpi.edu/sanlab/wiki/SANLab-CMhttp://cwl-projects.cogsci.rpi.edu/sanlab/wiki/SANLab-CM
Rensselaer CognitiveScience
Activity Networks
• Activity networks are directed, acyclic graphs where each node in the graph represents some process in time, and connections between nodes represent execution order
• Used in PERT charts for project management
34
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Activity Networks
• Activity Network Modeling in Cognitive Science
• First applied to cognitive modeling by Schweickert (1978; 1980)
• John & Gray’s work with CPM-GOMS (Project Ernestine, Milliseconds Matter)
35
Schweickert, R. (1980). Critical-Path Scheduling Of Mental Processes In A Dual Task. Science, 209(4457), 704-706.
Schweickert, R. (1978). A critical path generalization of the additive factor method: Analysis of a Stroop task. Journal of Mathematical Psychology, 18(2), 105–139. John, B. E. (1990). Extensions of GOMS Analyses to Expert Performance Requiring Perception of Dynamic Visual and Auditory Information. Paper presented at the ACM CHI'90 Conference on Human Factors in Computing Systems, New York.
Gray, W. D., John, B. E., & Atwood, M. E. (1993). Project Ernestine: Validating a GOMS analysis for predicting and explaining real-world performance. Human-Computer Interaction, 8(3), 237–309.
Gray, W. D., & Boehm-Davis, D. A. (2000). Milliseconds Matter: An introduction to microstrategies and to their use in describing and predicting interactive behavior. Journal of Experimental Psychology: Applied, 6(4), 322–335.
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Activity Networks
• Activity Networks provide a network of dependencies with the longest path through that network being named the Critical Path
• That is, the shortest time an activity can complete is determined by the longest path through the network
36
Tuesday, April 6, 2010
Rensselaer CognitiveScience
What a Critical Path Analysis can Tell You: Beginning
37
System!
Operations!
!
Visual!
Perceptual!
Operations!
Aural!
!
!
Cognitive!
Operations!
!
!
R-hand!
L-hand!
Motor !
Operations!
Verbal!
!
EyeMovement
Proposed Workstation
System!
Operations!
!
Visual!
Perceptual!
Operations!
Aural!
!
!
Cognitive!
Operations!
!
!
R-hand!
L-hand!
Motor !
Operations!
Verbal!
!
EyeMovement
Current Workstation
operators removed from slack time
Gray, W. D., John, B. E., & Atwood, M. E. (1993). Project Ernestine: Validating a GOMS analysis for predicting and explaining real-world performance. Human-Computer Interaction, 8(3), 237–309.
Tuesday, April 6, 2010
Rensselaer CognitiveScience
What a Critical Path Analysis can Tell You: Ending
38
System!
Operations!
!
Visual!
Perceptual!
Operations!
Aural!
!
!
Cognitive!
Operations!
!
!
R-hand!
L-hand!
Motor !
Operations!
Verbal!
!
!
EyeMovement
Current Workstation
System!
Operations!
!
Visual!
Perceptual!
Operations!
Aural!
!
!
Cognitive!
Operations!
!
!
R-hand!
L-hand!
Motor !
Operations!
Verbal!
!
!
EyeMovement
Proposed Workstation
operators added to!
critical path
Gray, W. D., John, B. E., & Atwood, M. E. (1993). Project Ernestine: Validating a GOMS analysis for predicting and explaining real-world performance. Human-Computer Interaction, 8(3), 237–309.
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Stochastic
• Most cognitive activity network models use mean times for operations (i.e., constant time)
• Schweickert and colleagues point out that cognitive operations have variability and that this variability can result in multiple critical paths
39
Schweickert, R., Fisher, D. L., & Proctor, R. W. (2003). Steps toward building mathematical and computer models from cognitive task analyses. Human Factors, 45(1), 77–103.
Goldstein, W. M., & Fisher, D. L. (1992). Stochastic networks as models of cognition: Deriving predictions for resource-constrained mental processing. Journal Of Mathematical Psychology, 36(1), 129-145.
Goldstein, W. M., & Fisher, D. L. (1991). Stochastic networks as models of cognition: Derivation of response-time distributions using the order-of-processing method. Journal Of Mathematical Psychology, 35(2), 214-241.
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Constructing a very simple CPM-GOMS model in SANLab
• Parts
• Interleaving
• Stochasticity
• Comparison of very simple models
40
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Building a Preliminary CPM-GOMS ModelCPM-GOMS Templates
Perceive Simple Sound
Perceive Visual Information With Eye Movement
41
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Building a Preliminary CPM-GOMS ModelCut & Paste & String Together
42
Inserted dependency line
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Building a Preliminary CPM-GOMS ModelRun Model Once to Generate Critical Path
Red outlined boxes show the critical path: note that ALL operations are on the
critical path!
43
Estimate time = 670 ms
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Two Problems:
• First: Assumes seriality – all operations of subtask A are completed before subtask B can begin
• FALSE: at this level of analysis the human cognitive controller can interleave subtasks
44
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Building a Preliminary CPM-GOMS ModelInterleave Operators
Interleaved operators
Critical path no longer goes through ALL operators
45
Estimate time = 620 ms
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Two Problems
• First: Assumes seriality – all operations of subtask A are completed before subtask B can begin
• FALSE: at this level of analysis the human cognitive controller can interleave subtasks
• Second: Assumes constancy of performance time for individual operations as well as for the entire task
• FALSE: process times for human cognitive, perceptual, and motor operations are stochastic, not constant
46
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Building a Preliminary CPM-GOMS ModelInterleave Operators + Stochastic Operation Times
47
Gamma Distribution(randomly sampled on each
model run)
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Interleave Operators + Stochastic Operation Times, then Run the Model 10,000 times
Two critical paths!Different mean times!
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Two Critical Paths
49
Mean time: 635 ms
Mean time: 598 ms
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Very Simple Model: Summary
50
Interleaving Constant/Stochastic Critical PathPredicted
TimesNo
Interleaving Constant One 670 ms
Interleaving Constant One 620 ms
Interleaving Stochastic Average 628 ms
Interleaving Stochastic 79% 635 ms
Interleaving Stochastic 21% 598 ms
Tuesday, April 6, 2010
Rensselaer CognitiveScience
What Does Stochasticity Add?
• Created SANLab models for each of the original Project Ernestine models
• Ran the SANLab models 10,000 times each
• Compared the fit of the original constant time model versus same models with stochastic times
• Looked at the variability across critical paths
51
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Fit of Original Constant Models versus Stochastic Models
52
●
●●
●
●
●
●
●
●
●
●
●
●●
0 10 20 30 40
010
2030
40
Constant
Empirical time (s)
Pred
icte
d tim
e (s
)
cc01
cc02cc03
cc04
cc05
cc06
cc07
cc09
cc10
cc11
cc12
cc13
cc16cc18
R2 = 0.716
●
● ●
●
●
●
●
●
●
●
●
●●
●
0 10 20 30 40
010
2030
40
Stochastic
Empirical time (s)
Pred
icte
d tim
e (s
)
cc01
cc02 cc03
cc04
cc05
cc06
cc07
cc09
cc10
cc11
cc12
cc13cc16
cc18R2 = 0.741
Old Workstation
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Predicted Variability Across Critical Paths for 6 Call Types
53
●
●●
●
●●
●
●●●
●
●
●●
●●
●
●●
●
●●●●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●●●●
●
●●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
0 2 4 6 8 10 13 16 19 22 25 28
2200
024
000
2600
028
000
cc01
Critical Path Index
Dur
atio
n (m
s)
●
●
●
●
●
●●
●
●
●●●●●●
●
●
●
●
●
●
●●
●●●●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●●
●
●
●●
●
●●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●●
●●
●
●
●●
●●●●
●
●
●●
●
●
●
●●
●●●
●
●
●●
●
●●●
●●
●
●●
●
●●●●●●●
●●●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●●●
●
●●●●●●
●
●●●
●
●
●●●●
●
●
●
●●●
●
●
●
●
●
0 1 2 3 4 5 6 7 8 9 11 13 15
8000
1200
016
000
cc02
Critical Path Index
Dur
atio
n (m
s)
●●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●
●
●●●
●●●●●●
●
●
●
●
●
●
●●●●●●●●
●
●●●●●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●● ●●
●
●●●
●
●
●●
●
●
●● ●
●
●●
●●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
● ●●
●
●
●
0 4 8 12 17 22 27 32 37 42 47 52 57
1000
012
000
1400
016
000
cc03
Critical Path Index
Dur
atio
n (m
s)
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●●
●
●
●●
●
●●●●
●
●●●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●●
●
●
●
● ●
●
●
●●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●
●
●
●
●
●
●●
●
●●
●●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
0 2 4 6 8 10 12 14 16 18 20 22 24
1800
022
000
2600
0
cc04
Critical Path Index
Dur
atio
n (m
s)
●
●
●
●
●●●●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●●●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●●●
●●●
●
●
●●
●●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0 3 6 9 12 16 20 24 28 32 36 40 44
1400
018
000
2200
0
cc05
Critical Path Index
Dur
atio
n (m
s)
●●
●
●
●
●●●●●●
●
●●●●●
●●
●
●●●●
●
●
●
●
●
●
●●●●●
●
●
●●●
●
●●
●●●●
●
●
●
●●
●
●●●
●
●
●
●●●
●
●●●
●●●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●●●●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
0 5 11 18 25 32 39 46 53 60 67 74 81 88
8000
1000
012
000
1400
0
cc06
Critical Path Index
Dur
atio
n (m
s)
Critical Paths for Six Models
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Implications of Adding Variability to Model Runs
54
• How long does normal performance take?
• When stochasticity is added we gain predictions of the variability in normal performance
• Are we planning for normal variability? Or are we expecting a constancy in real-time safety-critical human performance that does not exist?
• The variability produced by SANLab is in the absence of stress, fatigue, or danger - variability would only increase under those circumstances
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: SANLab
• Ease of Use
• Prerequisite Knowledge: B
• No computer science, but some sophistication in cognitive science (Masters? PhD?)
• Building Models: B-
• Easy (drag and click interface)
• Large models can be tedious to build
• Tools are being added to facilitate comparisons of critical paths and other measures
55
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: SANLab
• Hammer versus Toolbox
• A hammer not a toolbox: F
• CPM-GOMS level of analysis
56
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: SANLab
• Time Scale: D+
• One Level
• But, can inspect the model at various levels (e.g., total performance time, operations on the critical path, number of critical paths and differences among them, etc)
57
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: SANLab
• Generative: D
• Minor edits are easy, but no true generative capability
• If environment changes model must be edited to reflect those changes
• Emergence: D
• No speed up in performance, no optimization of strategy selection, no new strategies, no new declarative knowledge
• Can discover new Critical Paths
• Key feature of SANLab is the distinction it makes between variations on a single strategy (multiple critical paths) versus variations between different strategies
58
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card
59
Ease of Learning and
Use
Ease of Learning and
Use
Toolbox versus
Hammer
Time Scales Generative Emergence
Prereq Knwldg
Bldg Models
ACT-R D D+ C C B+ mixed
CogTool A+ A+ F F D F
SANLab B B- F D+ D D
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Outline
• Discussion of problems with existing technologies for computational cognitive modeling
• Introduction of three very different tools
• Discussion of their interconnectedness
• The shape of things to come
60
Tuesday, April 6, 2010
Rensselaer CognitiveScience
CogTool to ACT-R
• CogTool to ACT-R
• CogTool derives its KLM predictions from a simplified version of ACT-R
• Running a CogTool model creates a simple ACT-R model that is run to generate an ACT-R trace. The KLM times are based on this trace
• Good use of harnessing the power of ACT-R to generate a simplified model
• It may be possible to use CogTool to generate much of the repetitive detail in an ACT-R model & then allow the modeler to concentrate on the rest of the model
61
Tuesday, April 6, 2010
Rensselaer CognitiveScience
ACT-R to SANLab
• It is possible to import the trace of an ACT-R model into SANLab (Patton, 2009)
• This would allow you to vary the parameters (mean times and distribution) of one run of the model to determine how different settings would affect the outcome
• This would also alleviate much of the tedium of building SANLab models
62Patton, E. (2009). Revisiting Project Ernestine: Automated Building of CPM-GOMS Models from ACT-
R Model Traces. Rensselaer Polytechnic Institute, Troy.
Tuesday, April 6, 2010
Rensselaer CognitiveScience
CogTool to ACT-R to SANLab
• It is possible (Patton, 2010) to export the ACT-R trace of a CogTool model into SANLab
63
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Status of Tools
• Publicly Available:
• ACT-R: http://act-r.psy.cmu.edu/
• Cogtool: http://cogtool.hcii.cs.cmu.edu/
• SANLab: http://cwl-projects.cogsci.rpi.edu/sanlab/wiki/SANLab-CM
• The ACT-R to SANLab connection is experimental and is not released
• The CogTool to SANLab (via ACT-R) connection is also experimental and not released
64
Tuesday, April 6, 2010
http://act-r.psy.cmu.eduhttp://act-r.psy.cmu.eduhttp://cogtool.hcii.cs.cmu.eduhttp://cogtool.hcii.cs.cmu.edu
Rensselaer CognitiveScience
Outline
• Discussion of problems with existing technologies for computational cognitive modeling
• Introduction of three very different tools
• Discussion of their interconnectedness
• The shape of things to come
65
Tuesday, April 6, 2010
Rensselaer CognitiveScience
CogTool ⇔ [ACT-R] ⇔ SANLab
• Imagine two types of users
• Those who harness the power of ACT-R to go from CogTool to SANLab, but who do not know ACT-R
• Those who freely use the power of the entire system to facilitate their work on each of these three
• How might the combination of these three systems score on our criteria?
66
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: CogTool ⇔ ACT-R ⇔
67
ACT-R CogTool SANLab CogTool ⇔ SANLabCogTool ⇔ SANLab
⇔ ACT-R ⇔
Prerequisite Knowledge
D A+ B
B
MS in CogSci
D
Minor in CompSci, MS in CogSci
Building Models
D+ A+ B-
A+
Most SANLab models can be started in CogTool
B
If can export to ACT-R from both SANLab and
CogTool
• Ease of Use
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: CogTool ⇔ ACT-R ⇔
68
ACT-R CogTool SANLab CogTool ⇔ SANLabCogTool ⇔ SANLab
⇔ ACT-R ⇔
C F F C B
• Hammer versus Toolbox
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: CogTool ⇔ ACT-R ⇔
69
ACT-R CogTool SANLab CogTool ⇔ SANLabCogTool ⇔ SANLab
⇔ ACT-R ⇔
C F D+ C B
• Time Scales
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: CogTool ⇔ ACT-R ⇔
70
ACT-R CogTool SANLab CogTool ⇔ SANLabCogTool ⇔ SANLab
⇔ ACT-R ⇔
B+ D D D B+
• Generative
Tuesday, April 6, 2010
Rensselaer CognitiveScience
Report Card: CogTool ⇔ ACT-R ⇔
71
ACT-R CogTool SANLab CogTool ⇔ SANLabCogTool ⇔ SANLab
⇔ ACT-R ⇔Speed up with
practiceA F F F A
Optimized Strategy Selection
A F F F A
New Strategies F F F F F
New Declarative Knowledge
C F F F C
New Critical Paths
F F A A A
• Emergence
Tuesday, April 6, 2010
Rensselaer CognitiveScience
The Shape of Things to Come???
72
• Conclusions
• A small toolbox
• Far from complete
• But . . . the combination of the three is more powerful than any one by itself
• Maybe, this is the shape of things to come . . .
• Clearly need more work on integrating these tools, adding other tools to this toolkit, and creating different toolkits that have different sets of tools
Tuesday, April 6, 2010
Rensselaer CognitiveScience
The Shape of Things to Come
• Problems in making the vision happen
• CogTool has been guided by a vision of building cognitive science into the tools used by practitioners
• SANLab has been guided by a desire to build an easy to use tool to do a certain type of cognitive modeling
• ACT-R focus on theory not on applications
73
Tuesday, April 6, 2010
Rensselaer CognitiveScience
The Shape of Things to Come
• Lack of funding for tool building has hampered development
• SANLab - rests almost entirely on the programming skills of one brilliant undergraduate who is now a brilliant graduate student
• Little academic reward for building CogTool or SANLab
• Some funding support for CogTool, none for SANLab
• The synergy discussed here is a pleasant by-product but was not anticipated by the creators
74
Tuesday, April 6, 2010
Rensselaer CognitiveScience
The Shape of Things to Come
• Together the three are greater than the sum of their parts
• Ease of building models greatly increases if simple ACT-R models can be created from CogTool and SANLab models
• Combining any two of these helps alleviate the hammer problem by creating a small toolbox
• Alleviate the time scale issue
• Generatively & Emergence are not synergistic over tools, but simply enable CogTool and SANLab models to benefit from ACT-R’s generally higher scores on these attributes
75
Tuesday, April 6, 2010
Thank You!
Tuesday, April 6, 2010
Top Related