The Growth of Cognitive modeling The Growth of Cognitive modeling in Human-Computer Interaction in Human-Computer Interaction

Since GOMSSince GOMS

Presented by:Presented by:Daniel Loewus-DeitchDaniel Loewus-Deitch

Douglas GrimesDouglas Grimes

Purpose of Article (1)Purpose of Article (1)

Evaluate the current status of our Evaluate the current status of our methods for modeling HCI cognition methods for modeling HCI cognition and analyzing task performance.and analyzing task performance.

Review the evolution of GOMS and Review the evolution of GOMS and MHP and how subsequent research MHP and how subsequent research has supported and extended this has supported and extended this framework. framework.

Purpose of Article (2)Purpose of Article (2)

Examine 3 new directions for this Examine 3 new directions for this framework:framework: Study of learning and transfer.Study of learning and transfer. Study of errors.Study of errors. Analysis of parallel processes.Analysis of parallel processes.

Some Central IssuesSome Central Issues

How people transfer between skilled How people transfer between skilled performance and problem solving.performance and problem solving.

Designing consistent user interfaces.Designing consistent user interfaces. How people produce and manage How people produce and manage

errors.errors. How we interpret visual displays.How we interpret visual displays. When processes are parallel and When processes are parallel and

when they are serial.when they are serial.

Usefulness and Goals of Cognitive Usefulness and Goals of Cognitive ModelingModeling

Main goal:Main goal: Predicting how users will interact with Predicting how users will interact with

proposed designs.proposed designs.

Usefulness and Goals of Cognitive Usefulness and Goals of Cognitive ModelingModeling

Useful in:Useful in: Initially constraining the design space.Initially constraining the design space. Answering specific design decisions, Answering specific design decisions,

including various tradeoffs.including various tradeoffs. Estimating the total time for task Estimating the total time for task

performance.performance. Estimate training time and guide Estimate training time and guide

training documentation.training documentation. Discovering the most resource-intensive Discovering the most resource-intensive

and error-prone stages of an activity.and error-prone stages of an activity.

GOMS and MHPGOMS and MHP

Card et al. proposed their framework Card et al. proposed their framework to help system designers to help system designers Gather detailed knowledge about the Gather detailed knowledge about the

processes of perception to action.processes of perception to action. Generate predictions about human Generate predictions about human

behavior and task performance in “real, behavior and task performance in “real, naturalistic settings.”naturalistic settings.”

GOMS and MHPGOMS and MHP

Two key components:Two key components: MHP (Model Human Processor) – a MHP (Model Human Processor) – a

general characterization of the human general characterization of the human as an information-processing system, as an information-processing system, including aincluding a

System architectureSystem architecture Library of quantitative parameters that Library of quantitative parameters that

breaks down task performance into specific breaks down task performance into specific components.components.

GOMS and MHPGOMS and MHP

Two key components:Two key components: GOMS (Goals, Operators, Methods, and GOMS (Goals, Operators, Methods, and

Selecction rules) – A family of models Selecction rules) – A family of models that describes what the user needs to that describes what the user needs to know in order to perform computer-know in order to perform computer-based tasks.based tasks.

GOMS and MHPGOMS and MHP

Steps of theoretical process:Steps of theoretical process: User perceives activity on screen.User perceives activity on screen. Evaluates whether it is what is expected.Evaluates whether it is what is expected. Sets up an intention of the next step (goals).Sets up an intention of the next step (goals). Retrieves way to enact this intent on the Retrieves way to enact this intent on the

system.system. Executes appropriate motor movements.Executes appropriate motor movements. Repeat process.Repeat process.

Critical Steps of User ActivitiesCritical Steps of User Activities

AssumptionsAssumptions

Routine cognitive skills can be Routine cognitive skills can be described as a serial sequence of described as a serial sequence of cognitive operations and motor cognitive operations and motor activities.activities.

Each time parameter is independent Each time parameter is independent of context (the same in any task).of context (the same in any task).

Empirical data derived from use of Empirical data derived from use of text editors, graphic systems, etc. is text editors, graphic systems, etc. is generalizable.generalizable.

StrengthsStrengths

Can accurately predict the time it takes a Can accurately predict the time it takes a skilledskilled user to execute a task based on a set user to execute a task based on a set of composite actions.of composite actions.

Certain component parameters have been Certain component parameters have been found to be very consistent across various found to be very consistent across various tasks and stable across repeated tasks and stable across repeated experiments.experiments. KeystrokeKeystroke PointingPointing Moving handsMoving hands Retrieving a chunk of info. from LTM.Retrieving a chunk of info. from LTM.

StrengthsStrengths

Allows comparisons of different Allows comparisons of different design alternatives.design alternatives.

LimitationsLimitations

Generalizable to new domains?Generalizable to new domains? Applied only to skilled users.Applied only to skilled users. No account of learning or delayed recall.No account of learning or delayed recall. No account of errors (predicts perfect No account of errors (predicts perfect

performance).performance). Cognitive processes treated in same Cognitive processes treated in same

manner as perceptual and motor manner as perceptual and motor activities.activities.

No consideration of parallel processes.No consideration of parallel processes.

LimitationsLimitations

No consideration of mental workload.No consideration of mental workload. Does not determine functionality.Does not determine functionality. Does not address user fatigue.Does not address user fatigue. No account for individual differences.No account for individual differences. Does not predict user satisfaction or Does not predict user satisfaction or

acceptance.acceptance. Does not consider CSCW issues.Does not consider CSCW issues.

Advances in Modeling Specific Advances in Modeling Specific Serial Components (1)Serial Components (1)

Direct tests of two assumptions:Direct tests of two assumptions: Serial processingSerial processing Consistency of time parameters across Consistency of time parameters across

taskstasks Helped define and expand the MHP.Helped define and expand the MHP. Determined by studies involving Determined by studies involving

entering editor commands with entering editor commands with keyboard commands and entering keyboard commands and entering formulas in spreadsheets.formulas in spreadsheets.

Advances in Modeling Specific Advances in Modeling Specific Serial Components (2)Serial Components (2)

Parameters are grouped into 3 Parameters are grouped into 3 classes:classes: Motor movementMotor movement PerceptionPerception Memory and cognitionMemory and cognition

Parameters can be mapped onto the Parameters can be mapped onto the critical steps of the user activity critical steps of the user activity process (previous figure).process (previous figure).

Motor MovementsMotor Movements

KeyingKeying 230 msec230 msec Can vary with skill level of typist, Can vary with skill level of typist,

frequency of particular key use, and frequency of particular key use, and predictability of text being typed.predictability of text being typed.

Moving a mouseMoving a mouse 1100 msec per selection1100 msec per selection Varies with distance of movement and Varies with distance of movement and

size of target (Fitt’s Law).size of target (Fitt’s Law). Always a constant time of 1 s to began Always a constant time of 1 s to began

moving.moving.

Motor MovementsMotor Movements

Hand MovementsHand Movements Move from keyboard to pointing device.Move from keyboard to pointing device. Large-muscle movement.Large-muscle movement. Variation among different pointing Variation among different pointing

devices.devices.

PerceptionPerception

Perceptual processor is estimated to be 100 Perceptual processor is estimated to be 100 msec.msec.

Saccade is estimated at 230 msec.Saccade is estimated at 230 msec. ““Scanning” parameter, identified by Olson and Scanning” parameter, identified by Olson and

Nielson, found in experiment to be 2300 msec.Nielson, found in experiment to be 2300 msec. User scans spreadsheet screen, looking for cell User scans spreadsheet screen, looking for cell

addresses.addresses. Composite of scanning, storing, and retrieving.Composite of scanning, storing, and retrieving. Roughly supports previously identified time Roughly supports previously identified time

parameter estimate when broken down into parameter estimate when broken down into components.components.

Memory and Cognitive ProcessesMemory and Cognitive Processes

Memory retrievalMemory retrieval M (“mental”) = time to retrieve the next M (“mental”) = time to retrieve the next

unit of information from LTM.unit of information from LTM. Estimated at 1350 msec.Estimated at 1350 msec. On repeated trials, retrieval times drop On repeated trials, retrieval times drop

50% and remain flat, but keying times 50% and remain flat, but keying times remain the same.remain the same.

Memory and Cognitive ProcessesMemory and Cognitive Processes

Executing steps in a taskExecuting steps in a task GOMS provides an explicit GOMS provides an explicit

representation of the mental steps representation of the mental steps involved.involved.

Kieras and Polson programmed the Kieras and Polson programmed the procedures in production system procedures in production system formalism.formalism.

Memory and Cognitive ProcessesMemory and Cognitive Processes

Choosing among methodsChoosing among methods MHP assumes that more choices for a MHP assumes that more choices for a

response lead to longer response times.response lead to longer response times. Research suggests that a choice Research suggests that a choice

between methods is a complex cognitive between methods is a complex cognitive task, requiring a number of cognitive task, requiring a number of cognitive steps.steps.

Summary of cognitive engineering Summary of cognitive engineering parameters derived:parameters derived:

Enter a keystrokeEnter a keystroke 230 msec230 msec

Point with a mousePoint with a mouse 1500 msec1500 msec

Move hands to mouseMove hands to mouse 360 msec360 msec

PerceivePerceive 100 msec100 msec

Make a saccadeMake a saccade 230 msec230 msec

Retrieve from Retrieve from memorymemory

1200 msec1200 msec

Execute a mental Execute a mental stepstep

70 msec70 msec

Choose among Choose among methodsmethods

1250 msec1250 msec

Example of Applied Use of GOMSExample of Applied Use of GOMS

Walker et al. demonstrated how Walker et al. demonstrated how GOMS can help a designer narrow GOMS can help a designer narrow down his search.down his search.

Goal was to shorten menu-selection Goal was to shorten menu-selection time with nested menus.time with nested menus.

Example of Applied Use of GOMSExample of Applied Use of GOMS

3 adjustments were made and tested 3 adjustments were made and tested using GOMS techniques:using GOMS techniques: Pop up submenus on right rather than Pop up submenus on right rather than

bottom (shorten total distance user bottom (shorten total distance user much move cursor to make a selection).much move cursor to make a selection).

Target size grows as the distance from Target size grows as the distance from the cursor increases (Fittsized menus).the cursor increases (Fittsized menus).

Add a virtual border around the pop-up Add a virtual border around the pop-up menu (increase target size).menu (increase target size).

Accuracy in Predicting Composite Accuracy in Predicting Composite PerformancePerformance

Young and MacLean measured the times Young and MacLean measured the times of people entering a block of values in a of people entering a block of values in a spreadsheet using 2 different methods:spreadsheet using 2 different methods: Mouse methodMouse method Menu methodMenu method

Showed that using these parameters can Showed that using these parameters can provide sufficient accuracy for this level provide sufficient accuracy for this level of analysis (14% error range in this of analysis (14% error range in this case).case).

Extensions of Basic FrameworkExtensions of Basic Framework

Task-Action Grammar (TAG)Task-Action Grammar (TAG) Knowledge a user must have in order to Knowledge a user must have in order to

translate from goals to actions in a translate from goals to actions in a particular system.particular system.

Predicts learning with the relationship Predicts learning with the relationship between system features and natural between system features and natural world associations.world associations.

Consists of commands, features of goal, Consists of commands, features of goal, dictionary of tasks, and rules that dictionary of tasks, and rules that translate goals into actions.translate goals into actions.

Extensions of Basic FrameworkExtensions of Basic Framework

Production systemsProduction systems Represent GOMS structure and aspects Represent GOMS structure and aspects

of MHP.of MHP. Makes underlying knowledge much Makes underlying knowledge much

more explicit.more explicit.

Learning and TransferLearning and Transfer

Reaction to GOMS’ narrow focus on Reaction to GOMS’ narrow focus on skilled performance.skilled performance.

Time to learn has been advanced byTime to learn has been advanced by Cognitive Complexity TheoryCognitive Complexity Theory Payne and Green’s research on grammar Payne and Green’s research on grammar

rules.rules. Transfer of training has been advanced Transfer of training has been advanced

byby Kieras and Polson’s production system Kieras and Polson’s production system

modelsmodels

Kieras and Polson’s Kieras and Polson’s Cognitive Cognitive Complexity Theory Complexity Theory (1)(1)

Made advances by focusing onMade advances by focusing on Time to learn new procedures.Time to learn new procedures. Transfer of training between procedures.Transfer of training between procedures.

Used specialized language called Used specialized language called NGOMSL to facilitate the NGOMSL to facilitate the programming of production system programming of production system representations.representations.

Kieras and Polson’s Kieras and Polson’s Cognitive Cognitive Complexity Theory Complexity Theory (2)(2)

First determined number of steps in First determined number of steps in procedure and then assessed the procedure and then assessed the time it takes a person to learn the time it takes a person to learn the procedure.procedure.

Can’t generalize quantified learning Can’t generalize quantified learning times to naturalistic settings.times to naturalistic settings.

Payne and GreenPayne and Green

Number of rules determines ease of Number of rules determines ease of learning.learning. More often a rule can be used, the more More often a rule can be used, the more

consistent a system is.consistent a system is. More critical is whether features of More critical is whether features of

rules follow real-world rules already rules follow real-world rules already familiar to the user.familiar to the user.

Kieras and Polson’s Production Kieras and Polson’s Production System ModelsSystem Models

Makes explicit exactly what it is that a Makes explicit exactly what it is that a person has to learn to use a new system.person has to learn to use a new system.

Productions = units of learning.Productions = units of learning. Number of productions the two systems Number of productions the two systems

share is a good prediction of transfer.share is a good prediction of transfer. Makes consistency of design across Makes consistency of design across

systems explicit and quantifiable.systems explicit and quantifiable. TAG has some similar potential.TAG has some similar potential.

(Time to Learn Scatter Plot – (Time to Learn Scatter Plot – Poulson)Poulson)

Analysis of ErrorsAnalysis of Errors

Hypothesis: When working memory Hypothesis: When working memory is overloaded, errors will increases.is overloaded, errors will increases.

Test: Calculate Profit = Revenues – Test: Calculate Profit = Revenues – Costs, using Lotus 1-2-3 & IFPSCosts, using Lotus 1-2-3 & IFPS

Result: IFPS Result: IFPS burdens working burdens working memory memory fewer errors fewer errors

Similar studies on SQL queries.Similar studies on SQL queries.

Parallel ProcessesParallel Processes

Earlier cognitive studies assumed Earlier cognitive studies assumed serial processing.serial processing.

Skilled motor and perceptual action is Skilled motor and perceptual action is highly parallel:highly parallel: Expert typist, musician, athlete, driver, …Expert typist, musician, athlete, driver, …

Serial models overestimate time for Serial models overestimate time for parallel tasksparallel tasks

Skilled action is more parallel; novice Skilled action is more parallel; novice action more serial.action more serial.

Critical Path Analysis: Critical Path Analysis: Parallel + SequentialParallel + Sequential

Theory: 3 parallel processors: Theory: 3 parallel processors: perceptual, cognitive, and motorperceptual, cognitive, and motor

Studies of skilled typing and skilled Studies of skilled typing and skilled menu use. menu use. (Author’s interpretation, not (Author’s interpretation, not speaker’s):speaker’s): Expert (~33 wpm) limited by cognitive Expert (~33 wpm) limited by cognitive

processor (e.g., reading speed for typist)processor (e.g., reading speed for typist) Novice (~6 wpm) limited by motor Novice (~6 wpm) limited by motor

speed.speed.

Q’s on Critical Path AnalysisQ’s on Critical Path Analysis If cognitive processor is the bottleneck If cognitive processor is the bottleneck

for expert typist, why can expert typists for expert typist, why can expert typists read much faster than they type? read much faster than they type? Do we Do we “chunk” letters together in preconscious processing, “chunk” letters together in preconscious processing, then “unchunk” them to type?then “unchunk” them to type?

Where do we draw the line between Where do we draw the line between perception and cognition? perception and cognition? Example shows Example shows reading is one word at a time with reading is one word at a time with constant timeconstant time per per word, but recognizing words is a cognitive process, word, but recognizing words is a cognitive process, whether preconscious or not. Also ocular-motor skill.whether preconscious or not. Also ocular-motor skill.

Doesn’t address looping/cascading.Doesn’t address looping/cascading. In spite of Q’s In spite of Q’s more accurate more accurate

predictions for some tasks than purely predictions for some tasks than purely serial models.serial models.

What cognitive modeling (serial & What cognitive modeling (serial & parallel) has not revealed parallel) has not revealed

How we move from skilled to unskilled How we move from skilled to unskilled tasks smoothly throughout the day.tasks smoothly throughout the day.

Learning: Learning: “Athough we know that consistency in a “Athough we know that consistency in a system may make that system easier to learn … and system may make that system easier to learn … and easier to operate…, we do not know much more than easier to operate…, we do not know much more than that.”that.”

Errors: More than overloading working Errors: More than overloading working memorymemory

Individual differences in HCIIndividual differences in HCI Fatigue and other stress factors: machines Fatigue and other stress factors: machines

don’t tire or get stressed out; people do.don’t tire or get stressed out; people do.

From GOMS to GamesFrom GOMS to Games Prior studies based on static visual Prior studies based on static visual

display of text and numbers.display of text and numbers. Bonnie John & Alonso Vera extended Bonnie John & Alonso Vera extended

GOMS model to more interactive GOMS model to more interactive HCI’s --video games, call handling, & HCI’s --video games, call handling, & browsing.browsing.

Subject: 9 year-old expertSubject: 9 year-old expert Result – 2 levels: Result – 2 levels:

GOMS predicts functional interaction wellGOMS predicts functional interaction well Keystroke accuracy ~ 50%Keystroke accuracy ~ 50%

Program (HI-SOAR) learns ~ humansProgram (HI-SOAR) learns ~ humans

ConclusionConclusion

GOMS and its successors are useful for GOMS and its successors are useful for many repetitive HCI tasks – bank tellers, many repetitive HCI tasks – bank tellers, ATM’s, airline reservations, ATM’s, airline reservations, bookkeeping… Cognitive modeling bookkeeping… Cognitive modeling saves $ + time saves $ + time less need to build and test prototypes on less need to build and test prototypes on

users when models can predict results.users when models can predict results. User testing still needed for reality check User testing still needed for reality check

– “situated cognition” = understanding – “situated cognition” = understanding tasks in their human and environmental tasks in their human and environmental context.context.

(Start of extra slides)(Start of extra slides)

Learning Computer TasksLearning Computer Tasks

Limitation of GOMS: Restricted to Limitation of GOMS: Restricted to skilled performance. Didn’t address skilled performance. Didn’t address learning.learning.

Cognitive Complexity Theory Cognitive Complexity Theory (Kieras & Polson)(Kieras & Polson)

2 aspects of their learning model:2 aspects of their learning model: Time to learn a new system or taskTime to learn a new system or task Transferring knowledge between tasksTransferring knowledge between tasks

Time to LearnTime to Learn

Kieras developed a high-level Kieras developed a high-level programming language to describe programming language to describe learning simple computer tasks, like learning simple computer tasks, like data entry or writing an SQL query.data entry or writing an SQL query.

Keiras & Polson found learning each Keiras & Polson found learning each step of a task takes ~ 30 s. Other step of a task takes ~ 30 s. Other studies found shorter times.studies found shorter times.

Current best guess ~ 25 s./step for Current best guess ~ 25 s./step for learning.learning.

Learning & ComplexityLearning & Complexity

Number of rules to learn is not as Number of rules to learn is not as important how well the rules follow important how well the rules follow real-world features the learner real-world features the learner already knows.already knows.

Implication: Metaphors work, as long Implication: Metaphors work, as long as they are consistent.as they are consistent.

Learning Computer Tasks (1): Learning Computer Tasks (1): TheoryTheory

K & P’s production rules make K & P’s production rules make explicit exactly what a person has to explicit exactly what a person has to learn to master a new system.learn to master a new system.

Hypothesis: If # of productions Hypothesis: If # of productions (~steps) is a measure of how long it (~steps) is a measure of how long it takes to learn, then 2 systems with takes to learn, then 2 systems with same # of productions should take same # of productions should take same time to learn.same time to learn.

Learning Computer Tasks (2): Learning Computer Tasks (2): StudiesStudies

Example: Compare time it takes to Example: Compare time it takes to learn:learn: How to copy a floppy How to copy a floppy How to print a documentHow to print a document

Result: Learning times are consistent Result: Learning times are consistent across systems.across systems.

Value: When designing systems, favor Value: When designing systems, favor alternatives with simpler production rulesalternatives with simpler production rules

(easier to learn!)(easier to learn!)

Philosophical Critique of GOMS Philosophical Critique of GOMS

Does it reduce human to machine?Does it reduce human to machine? Answer: Not intended to address Answer: Not intended to address

philosophy. Only intended to aid in philosophy. Only intended to aid in system design.system design.

Bottom line: Predictions average Bottom line: Predictions average within 14% of observed values for a within 14% of observed values for a narrow range of serial tasks.narrow range of serial tasks.

Practical Critiques of GOMS Practical Critiques of GOMS

Restricted to skilled performanceRestricted to skilled performance Doesn’t predict learningDoesn’t predict learning Doesn’t apply to parallel tasksDoesn’t apply to parallel tasks

Extensions to Basic GOMS FrameworkExtensions to Basic GOMS Framework

Purpose: Less restrictive framework Purpose: Less restrictive framework for understanding HCI.for understanding HCI.

3 areas of analysis since GOMS:3 areas of analysis since GOMS:1.1. Learning Learning

2.2. ErrorsErrors

3.3. Parallel processesParallel processes

Cognitive Engineering Cognitive Engineering GrammarsGrammars

Purpose: Define cognitive tasks in Purpose: Define cognitive tasks in terms of countable components for terms of countable components for production rules.production rules.

Example: Task-Action Grammar (TAG)Example: Task-Action Grammar (TAG) Commands, e.g. “Left-click,” “press Commands, e.g. “Left-click,” “press

<enter>”<enter>” Features of goal, e.g. direction of cursorFeatures of goal, e.g. direction of cursor Tasks, e.g., “Add 3 numbers.”Tasks, e.g., “Add 3 numbers.” Rules to translate goals into action.Rules to translate goals into action.