The HumanProcessing and Memory

Human Computer Interaction, 2nd Ed.

Dix, Finlay, Abowd, and Beale

Chapter 1

Model Human Processor + AttentionRecall, “purely and engineering abstraction”

• Sensory store– Rapid decay “buffer” to hold

sensory input for later processing

• Perceptual processor– Recognizes symbols, phonemes– Aided by LTM

• Cognitive processor– Uses recognized symbols– Makes comparisons and

decisions– Problem solving– Interacts with LTM and WM

• Motor processor– Input from cog. proc. for action– Instructs muscles– Feedback

• Results of muscles by senses

• Attention– Allocation of resources

Overview

• Will look at elements of human information processing from a slightly different orientation than “engineering abstraction”

• A bit more fine grained analysis, following from psychological studies– But, it is these psychological studies from which the “engineering

abstraction” is derived

• 3 stage model of human memory– Iconic buffer, STM, LTM

• Models of LTM

• Reasoning

• Problem solving

Model Human Processor + AttentionRecall, “purely and engineering abstraction”

• Sensory store– Rapid decay “buffer” to hold

sensory input for later processing

• Perceptual processor– Recognizes symbols, phonemes– Aided by LTM

• Cognitive processor– Uses recognized symbols– Makes comparisons and

decisions– Problem solving– Interacts with LTM and WM

• Motor processor– Input from cog. proc. for action– Instructs muscles– Feedback

• Results of muscles by senses

• Attention– Allocation of resources

3-Stage Model of Human Memory

• Sensory (here, iconic) memory – “very” short term memory– lasts 1-2 seconds, infinite capacity

• Short-term memory (Working memory)– lasts ~ 18 seconds, holds 1.75 (7+/-2 items)

• Long-term memory– infinite capacity; short of damage is permanent– Recall vs. Recognition (Remember vs. Know)

• Retrieval cues

• Will demonstrate later in class … http://www.if.uidaho.edu/~marbjm/class%202.pdf

“Executive” - Attention

• Central “executive” controls tasking – Pays, or allocates, attention– Bandwidth of attention is limited

• Tasks that require the same resources interfere with one another

• Attention is both a low-level and high-level property of visionhttp://www.if.uidaho.edu/~marbjm/class%202.pdf

Sensory Memory: “Very” Short Term Memory

• Sensory buffers for stimuli received through senses– iconic memory: visual stimuli– echoic memory: aural stimuli– haptic memory: tactile stimuli

• Examples– “sparkler” trail– stereo sound

• Continuously overwritten – demo follows

A Test – of Visual Iconic Memory

• Will present figure briefly (~1/2 second)

• Try to remember as many elements as you can

• Write them down

The Phenomenon

• After presentation, did you continue to “see” the items?

– Some purely physiological based “seeing”:• Afterimage• Bleaching of pigments• “bright, or colored, stuff”

– But also, there is a more “memory-based” image (process further downstream in memory system)

• Iconic memory• “dark, or veridical, stuff”• Reading from the iconic buffer

Reading from the Iconic Buffer, 1

• Typically can list 3 – 7 items named

• Short lived visual, or iconic, buffer– holds the image for a second or two

• Read images and place in STM– 3-stage model

• Can get about 5-7 items until run out of short term (working) memory capacity

• Limitation of 5-7 comes from:– Decay of iconic memory– Rate can read from visual buffer– Capacity of working memory

• In each fixation between saccadic eye movements, image of world captured

Set of miscellaneous symbols

a

VisualSearch orMonitoringStrategy

EyeMovementControl

Useful VisualField of View

Reading from the Iconic Buffer, 2

• Again, Limitation of 7 comes from:– Decay of iconic memory– Rate can read from visual buffer– Capacity of working memory

• From each image, – brain must identify objects, – match them with objects previously

perceived, and – take information into working memory for

symbolic analysis

• Search light model of attention (for vision) – Visual information is acquired by pointing

fovea at regions of visual field that are interesting

– Then using a scanning process in which objects are read from an image buffer from more extensive processing

Set of miscellaneous symbols

a

VisualSearch orMonitoringStrategy

EyeMovementControl

Useful VisualField of View

Attention

• Spotlight metaphor– Spotlight moves serially from one input channel to another– Can focus attention (and perceptual processor) on only one input channel at a

time• Location in visual field, voice in auditory field, …, anything

• Visual dominance: – Easier to attend to visual channels than auditory channels

• All stimuli within spotlighted channel are processed in parallel– Whether you want to or not– Can cause “interference” - demo

Say the Colors of the Words

• Easy enough – didn’t take too long

Say the Colors of the Words

• Took longer … Stroop effect• For design:

– Choose secondary characteristics of display to reinforce message

Again, Human Memory Stages

• Sensory (here, iconic) memory– lasts 1-2 seconds, infinite capacity

• Short-term memory (Working memory)– lasts ~ 18 seconds, holds 1.75 (7+/-2 items)

• Long-term memory– infinite capacity; short of damage is permanent– Recall vs. Recognition (Remember vs. Know)

• Retrieval cues http://www.if.uidaho.edu/~marbjm/class%202.pdf

Short-term memory (STM)

• “Scratch-pad” (or buffer) for temporary recall– rapid access ~ 70ms– rapid decay ~ 200ms– limited capacity - 7± 2 chunks

• Chunking, recoding, etc. – affects amount of information retained, entering LTM

Example - Chunking

HEC ATR ANU PTH ETR EET

Long-term Memory (LTM)

• Repository for all our knowledge– slow access ~ 1/10 second– slow decay, if any– huge or unlimited capacity

• Episodic and semantic memory– Episodic (episodes): Serial memory of events– Semantic (“meanings”): Structured memory of facts, concepts, skills

• Also, procedural and declarative memory– “Processes” vs. “facts”

LTM – Models of Semantic Memory

• Semantic memory structure– Contains LTM knowledge of world– Provides access to information

– Generic knowledge -- specific details lost– Represents relationships between bits of information

– Important for rule-based behavior• Supports inference

• Many models, theories, accounts, schemata proposed

• Semantic network model (example next slide): – E.g., Inheritance – child nodes inherit properties of parent nodes– Relationships between bits of information explicit– Supports inference through inheritance

• Other Models (examples follow):– Scripts, frames, production rules

Early Model of Semantic MemoryCollins and Quillian

• Collins & Quillian’s Teachable Language Comprehender

• Semantic memory is organized as a network of interrelated concepts

• Each concept is represented as a node

• Concepts are linked together by pathways

• Economy of representation

• Activation of one concept spreads to interconnected nodes

• Remind you of anything from computer science?

Early Model of Semantic MemoryCollins and Quillian

• Collins & Quillian’s Teachable Language Comprehender

Early Model of Semantic MemoryCollins and Quillian

• Spreading Activation

• Working memory is activated LTM

• When a concept becomes active, activation spreads to all other interconnected nodes

• Activation spreads to all related nodes

• How do you evaluate sentences like “Is a robin is an animal”?

Early Model of Semantic MemoryCollins and Quillian

• Spreading Activation

• Activation spreads from each of the concept nodes (Robin & Animal)

• When two spreading activations meet, an intersection is formed

• Robins ==> BIRD <== Animals

• If no intersection, relatively fast no

• If intersection, decision stage operates to determine if sentence is valid

Is a robin an animal?

Tests of Spreading Activation

• Sentence verification task– Time to respond yes or no

• Takes time for activation to spread

• Greater distances ==> longer RT

• Verification time for items 0, 1, and 2 links

But, it’s not that simple ...

• E.g. typicality effects - how many links separate:– A canary is a bird?– A robin is a bird?– A chicken is a bird?– An ostrich is a bird?

• But, RT varied - less typical birds took longer than more typical birds

Semantic Relatednes and Semantic Priming

• Semantic relatedness– Spreading activation between related concepts– Activation of one concept partially activates

semantically related concepts

• Semantic priming– Stimulus 1 ==> Stimulus 2– (Prime) ==> (Probe)– Test spreading activation by manipulating

semantic relationship between prime & probe

• Concepts linked by spreading activation• Prime: Probe:• Doctor Nurse• Bread Butter• Doctor Butter• Bread Nurse

• Sometimes prime facilitates processing

Semantic Relatednes

• Recall, semantic relatedness

– Spreading activation between related concepts

– Activation of one concept partially activates semantically related concepts

• So, can focus on relatedness, without explicitly indicating links

Spreading Activation - Fin

• Semantic priming commonplace– Can exploit in design– Indeed, in design can exploit all information about how human operates

• Spreading activation is thought to be automatic

• Governed by data-driven aspects of processing

• How do expectancies affect semantic access?– Automatic vs Conscious Strategies (attentional)– Fast vs Slow– Effortless vs Effortful– Benefits vs Costs & Benefits

Fyi – Another semantic network

Models of LTM – Frames, or Schemata

• Information organized in “memorial data structures”

• Schemata– Stored framework or body of knowledge

– Conceptual framework for interpreting information

– Biased information processing to relate new material to what we already know

– Alters way we perceive things

– Individual differences in perception and memory

• Frames– Slots in structure instantiated with values for

instance of data– Type–subtype relationships

DOG

Fixed legs: 4

Default diet: carniverous sound: bark

Variable size: colour

COLLIE

Fixed breed of: DOG type: sheepdog

Default size: 65 cm

Variable colour

Models of LTM - Scripts

• Model of stereotypical information required to interpret situation

• Script has elements that can be instantiated with values for context

Script for a visit to the vet

Entry conditions: dog illvet openowner has money

Result: dog betterowner poorervet richer

Props: examination tablemedicineinstruments

Roles: vet examines diagnoses treatsowner brings dog in pays takes dog out

Scenes: arriving at receptionwaiting in roomexaminationpaying

Tracks: dog needs medicinedog needs operation

Models of LTM - Production Rules

• Representation of procedural knowledge.

• Condition/action rules if condition is matched then use rule to determine action.

IF dog is wagging tail, THEN pat dog

IF dog is growling, THEN run away

LTM - Storage of information

• LTM much studied in psychology:

• Rehearsal– information moves from STM to LTM

• Total time hypothesis– amount retained proportional to rehearsal time

• Distribution of practice effect– optimized by spreading learning over time

• Structure, meaning and familiarity– information easier to remember

LTM - Forgetting

• Decay– information is lost gradually but very slowly

• Interference– new information replaces old: retroactive interference– old may interfere with new: proactive inhibition

• So, ... may not forget at all, memory is selective …!

• Also, affected by emotion – can subconsciously `choose' to forget

LTM - Retrieval

• Should be familiar from heuristics

• Recall– information reproduced from memory can be assisted by cues,

e.g. categories, imagery

• Recognition– information gives knowledge that it has been seen before– less complex than recall - information is cue

Thinking – Cognitive Processing

• Humans reason, process information, like, well, humans– Recall, any theory is an abstraction and, thus, captures some elements of

phenomenon, and misses others– Question is …

• Is the account (theory, model) useful in the context and for the purpose for which it is used?

• Basic forms of reasoning, or, forming inferences, are useful in understanding broad outlines of human cognition

– Deduction– Induction– Abduction

• Problem solving– Gestalt– Problem Space– Analogy– Skill acquisition

Reasoning

• Deduction:– derive logically necessary conclusion from given premises

• e.g., If it is Friday, then she will go to work - It is Friday, therefore she will go to work

– Logical conclusion not necessarily true:e.g., If it is raining, then the ground is dry - It is raining, therefore the ground is dry

• Induction:– Generalize from cases seen to cases unseen

• e.g., All elephants we have seen have trunks - therefore all elephants have trunks.

– Unreliable (but useful):• Can only prove false not true

• Abduction:– Reasoning from event to cause e.g., Sam drives fast when drunk. If I see Sam driving fast, assume drunk.– Unreliable:

• can lead to false explanations

FYI - Induction vs. Deduction

• Induction: Make observations first, then draw conclusions– Organized data survey (structured analysis,

visualization) of raw data provide basis for interpretation process

– Interpretation process will produce knowledge that is being sought

– Experience of individual scientist (observer) is crucial– Important: selection of relevant data, collection method,

and analysis method– Data mining is an important knowledge discovery

strategy• ubiquitious data collection, filtering, classification, and focusing is

crucial

• Deduction: Formulate hypothesis first, then test hypothesis– Via experiment and accept/reject– Data collection more targeted than in induction– Only limited data mining opportunities Mueller, 2003

Problem Solving

• Process of finding solution to unfamiliar task using knowledge– Complex, time consuming process

– Selections not immediately obvious– May require many steps– May involve insight– May use analogy– Solutions often counterintuitive– Several theories, or accounts

• Gestalt – Problem solving both productive and reproductive– Productive draws on insight and restructuring of problem– Attractive but not enough evidence to explain “insight” etc.– Move away from behaviourism and led towards information processing theories

• Others: Insight, Functional fixedness, Analogy

Problem Solving Cycle

• One schema – consider “task performance”

Insight

• Early - Kohler (a Gestalt psychologist) in Canary islands in WWI– Studied problem solving in chimpanzees– Sultan and the Banana:

• Learned how to get banana with longer pole• Then given shorter poles that wouldn’t reach• Flash of “insight”, Sultan put the poles together

• Sudden perception of useful or proper relations– Solutions will sometimes “spring to mind”– Pieces fall into place

• First attempts to solve don’t work– Production hindered by unwarranted assumptions.– Insight occurs when the assumption is removed

Strayer, Utah: http://www.psych.utah.edu/psych3120-classroom/

Problem solving (cont.)

• Problem space theory

– Problem space comprises problem states– Problem solving involves generating states using legal operators– Heuristics may be employed to select operators

e.g. means-ends analysis– Operates within human information processing system

e.g. STM limits etc.– Largely applied to problem solving in well-defined areas

e.g. puzzles rather than knowledge intensive areas

Problem solving (cont.)

• Analogy– Analogical mapping:

• novel problems in new domain?• use knowledge of similar problem from similar domain

– Analogical mapping difficult if domains are semantically different

• Skill acquisition – e.g., “expert” performance– Skilled activity characterized by chunking

• lot of information is chunked to optimize STM

– Conceptual rather than superficial grouping of problems– Information is structured more effectively

Individual Differences

• Long term– Sex, physical and intellectual abilities

• Short term– Effect of stress or fatigue

• Changing– Age

• Dix says ask: – Will design decision exclude section of user population?– (or, more generally) How does design differentially affect sections of the

population?– i.e., Universal Usability

The “Model Human Processor + Attention” isSimilar to Ware (2004) Model

• We’ll look at one more model of cognitive (and visual) processing

– All are in fact much the same, but focus on different goals

The “Model Human Processor + Attention” isSimilar to Ware (2004) Model

• We’ll look at one more model of cognitive (and visual) processing

– All are in fact much the same, but focus on different goals

• Card et al. model was developed in context of predicting user performance

– E.g., set parameters and perform simulation

• Ware’s model includes much the same elements, but focuses on those which are most relevant for processing of visual information in context of task performance

The “Model Human Processor + Attention” isSimilar to Ware (2004) Model

• Sensory store– Rapid decay “buffer” to hold

sensory input for later processing

• Perceptual processor– Recognizes symbols, phonemes– Aided by LTM

• Cognitive processor– Uses recognized symbols– Makes comparisons and

decisions– Problem solving– Interacts with LTM and WM

• Motor processor– Input from cog. proc. for action– Instructs muscles– Feedback

• Results of muscles by senses

• Attention– Allocation of resources

A Model of Perceptual ProcessingQuick Overview

• An information processing (the dominant paradigm) model– “Information” is transformed and processed

• Physical light does excite neurons, but at this “level of analysis” consider information– Gives account to examine aspects important to visualization

• Here, clearly, many neural subsystems and mapping of neural to ip is pragmatic– In spirit of visualization as evolving discipline, yet to develop its theories, laws, …

• Stage 1: Parallel processing to extract low-level properties of the visual science• Stage 2: Pattern perception• Stage 3: Sequential goal-directed processing

What we do is design information displays!

Stage 1: Parallel Processing to Extract Low-level Properties of Visual Scene

• (Very first) neurons fire

• Visual information 1st processed by

– large array of neurons in eye – primary visual cortex at back of brain

• Individual neurons selectively tuned to certain kinds of information

– e.g., orientations of edges or color of light

– Evoked potential experiments

• In each subarea large arrays of neurons work in parallel

– extracting particular features of environment (stimulus)

Stage 1: Parallel Processing to Extract Low-level Properties of Visual Scene

• At early stages, parallel processing proceeds involuntarily

– Largely independent of what choose to attend to (though not where look)

• Is rapid, – If want people to understand

information fast, should present in way so is easily detected by these large, fast computational systems in brain

• Stage 1 processing is:– Rapid and parallel– Entails extraction of features,

orientation, color, texture, and movement patterns

– “transitory”, only briefly held in iconic store

– Bottom up, data-driven

Stage 2: Pattern Perception

• Rapid processes

• Divide visual field into regions and simple patterns, e.g.,

– Continuous contours – Regions of same color – Regions of same texture– …

• “Active”, but not conscious processes

• Specialized for object recognition– Visual attention and memory

• E.g., for recognition must match features with memory

– Task performing will influence what perceived

– Bottom up nature of Stage 1, influenced by top down nature of Stage 3

Stage 2: Pattern Perception

• Specialized for interacting with environment

– E.g., tasks involving eye-hand coordination

• “Two-visual system hypothesis”– One system for locomotion and eye-hand

coordination• The “action system”

– One system for symbolic object manipulation

• The “what system”

• Characteristics:– Slow serial processing– Involvement of both working (vs. iconic)

and long-term memory– Both bottom up and top down

• More emphasis on arbitrary aspects of symbols than Stage 1

• Top-down processing– Different pathways for object recognition

and visually guided motion

Stage 3: Sequential Goal-Directed Processing

• At highest level of perception are the objects held in visual memory by demands of active attention

• To use an external visualization, we construct a sequence of visual queries that are answered through visual search strategies

• Only a few objects can be held at a time

• They are constructed from available patterns providing answers to the visual queries

Stage 3: Sequential Goal-Directed Processing

• They are constructed from available patterns providing answers to the visual queries

• E.g., if use a road map to look for a route, the visual query will trigger a search for connected red contours (representing major highways) between two visual symbols (representing cities)

• Are other subsystems, as well– Visual object identification process

interfaces with the verbal linguistic subsystems of the brain so that words can be connected to images

– The perception-for-action subsystem interfaces with the motor systems that contril muscle movements

End

• .