Sensory systems Chapter 50. Five senses Touch Taste Smell Sound Vision.
Sensory Channels and Media Systems: Vision, …fisher/SG03CourseIntersensory.pdfTHE UNIVERSITY OF...
Transcript of Sensory Channels and Media Systems: Vision, …fisher/SG03CourseIntersensory.pdfTHE UNIVERSITY OF...
THE UNIVERSITY OF BRITISH COLUMBIA
Sensory Channels and Media Systems: Vision, Hearing, & Haptics
Brian Fisher
Media and Graphics Interdisciplinary Centre
Fisher: Sensory Channels
Intersensory Interactions
• Intro and metacognitive gap
• Integrating cogsci theory with design
• Cognitive Architecture
– Modularity and multimodal interaction
• Information hiding-- conflict resolution
• Cognitive impenetrability
• Performance differences between modules
• Recalibration
– Spatial indexes in complex environments• Multimodal cue matching within modules
Fisher: Sensory Channels
Vision systems to multimodality
• Ron: Vision systems and subsystems– Pre-attentive vision (gist, layout, events)
– Attention (grab ~5 objects for processing)
– Combine for “virtual representation”
• Extend system concept to modalities– Some are similar across modalities
– Some are multimodal
– Some are task-dependent
Fisher: Sensory Channels
Key Points for HCI Practice
• The Metacognitive Gap: The need for a grounded Cognitive Science approach
• Reflective design practice methods– Integrating CogSci with interaction design– Iterative design cycle
• Examples of extended HCI – Cognitive architecture: Multimodal displays and
how they are understood – Situated cognition: embodied interaction with
complex displays
Fisher: Sensory Channels
Introduction
History -- why we need to adapt HCI methods for multimodal interaction
Fisher: Sensory Channels
HCI History• “Classic” HCI: User/Task/Tool Model
– Task, Protocol & GOMS/keystroke analysis
• Command-line, menus, workplace systems
• Theoretical underpinnings
– Cogsci of conscious thought
– Learning, Memory, Reasoning
– Sequences of operations
Fisher: Sensory Channels
Ergonomic HCI domain limits
TaskTool
User
What if I am doing this for fun?What if I want new insights?What if I want to communicate with someone?What if I am exploring a complex environment?
Fisher: Sensory Channels
Evolution towards multimodal displays
• WIMP interface: visual semantics– Metaphorical tool icons on desktop
– Direct manipulation
• Information visualization: Information processing in the visual system– Visual analogs of information
– Spatial Instruments
Fisher: Sensory Channels
Current development cycle
Walkthrough or experiment
Implement prototype
Design Assess
Craft model does not scale to large design space
Craft
Fisher: Sensory Channels
Future challenges
• Perception, cognition, and action in an immersive
multimodal environment populated with objects
events and actors
• Applications in entertainment, cognition,
communication
• Blend of virtual and real spaces… with seams
– Are the rules consistent?
– Can users shift between them?
– Can frames support rule shifts?
Fisher: Sensory Channels
Opportunities for creative design
• Environments: Affordances for exploration– Spatial cognition, human space constancy theory
• Support for creative & logical thinking – Problem solving, embodied cognition models
• Media-based communication & collaboration – Metacognition, distributed cognition
• Experience (Kansei) engineering: Moving beyond usability
Fisher: Sensory Channels
Effective Interface Design for Rich Sensory Environments
The interaction between display characteristics and the information processing characteristics of the
user’s perceptual, motor, and cognitive processes will largely determine interface performance
Fisher: Sensory Channels
Metacognitive Gap
• Intuitions about thoughts,goals and plans (“folk Psychology”) are reasonably accurate
• Intuitions about how people see, hear, and remember are very inaccurate
• Lack of awareness of the limits of intuition is the “Metacognitive gap”
• Design-by-intuition leads to bad user experience
Fisher: Sensory Channels
Evolving interaction design models
• Guidelines are (still) inadequate• User-centred design inadequate for
rich sensory environments• Need for theory-rich, evidence-based
approach: design for lower-level processes
• Must integrate with higher-order cognitive task analysis and user-centred design practices
Fisher: Sensory Channels
Bridging the theory/practice gap
• Convincing designers that there is something to
understand—the “metacognitive gap” of folk
Psychology
• Convincing Cognitive Scientists to answer relevant
questions—Complex data displays and multiple
tasks and the Psychophysics reductionist approach
• Integrating research and design—Finding a
common language
Fisher: Sensory Channels
Multimodal development cycle
Constrain large multimodal design space
Assess specific
aspects of interaction
Literature Foraging:
HCI & graphicsPsychologyKinesiologySociology
Architecture…
Design for key sensory & motor
systems
Walkthrough or experiment
Implement prototype
Fisher: Sensory Channels
Fields of interest: Experimental Psychology:
– Founded~ 100 years ago
– Areas of study• Psychophysics—Vision, hearing, tactile senses
• Attention—Endogenous, exogenous, sustained
• Learning and memory
• Goal is often information processing algorithms
• Discussed in Module 2
Fisher: Sensory Channels
Useful things from Psychology
• Perception and attention– Visual & auditory acuity & discrimination
– Colour perception
– Salient display changes, change blindness
• Learning and memory– Primacy, recency
– Skill acquisition
– STM limits
– State-dependent learning
• Decision making– Base-rate neglect
– End effects, biases
Fisher: Sensory Channels
Sound perception basics
• Auditory Psychophysics – Waves of air pressure, 16Hz - 20kHz.
– Sensitivity fits Stevens Power Law
– Frequency masking (used in compression)
• Hearing in a complex world– Auditory “streams”
– Auditory localization
– Acoustic communication
Fisher: Sensory Channels
Complex auditory scenes • Auditory stream segregation system
(Bregman)• Gestalt grouping rules
– Proximity:
– Similarity:
– …but in frequency and timbre spaces
http://tinyurl.com/i2hf
XXX XXX XXX
X X X O O O X X X
Fisher: Sensory Channels
Acoustic communication• Speech
– Highly rule-based
– Categorical perception
– Multimodal speech (visemes)
• Music– Somewhat rule-based
– Stream segregation
• Soundscape– Some regularity “Acoustic ecology”
– Spatial & temporal coherence w. vision, haptic
– Sound events lead us to search for their cause
Fisher: Sensory Channels
Spatial sound
• Left vs. right (ear)
– Relative intensity (high freq.)
– Onset time difference (high freq., rapid onset)
– Phase differences (low freq.)
• Azimuth, distance
– Room acoustics
– Pinna reflections/attenuations (HRTF)
• Spatial sound is evocative (Audium: Shaff)
Fisher: Sensory Channels
Fields of interest: Kinesiology & related disciplines
• Science of human movement – Neuroscience
– Mechanics
– Anthropometry
• Examples: Fitts’ law, GOMS/Keystroke
• Goal is often perceptually guided behaviour
Fisher: Sensory Channels
Haptic perception basics
• Cutaneous, tactile & somatic system
– Stimulus to the skin
– Heat, pressure, vibration, slip, pain
• Kinesthetic & proprioceptive system
– Limb position, motion, force
– Receptors in muscles, tendons, and joints
– Stimulated by bodily movements
Fisher: Sensory Channels
Cutaneous, tactile & somatic system
• Thermoreceptors: change in skin temp
• Mechanoreceptors: pressure, vibration, slip
• Nocioreceptors: pain
Systems within modalities: (similar structure to vision, audition)
But we still have access to some haptic “primitives”
Fisher: Sensory Channels
Haptic bandwidth of finger & hand
5000-10000 Hz : Sense vibration in manipulative task
320 Hz : Discriminate two consecutive force input signals
20-30 Hz : Min. force input for meaningful perception
12- 16 Hz : Can correct grasping forces if the object slips
8 - 12 Hz : Can correct for positional disturbances
5 - 10 Hz : Max comfortable force & motion commands
1 - 2 Hz: Max to react to unexpected force/position signal
Shimoga,1992
Fisher: Sensory Channels
Distribution of tactile sensors
Fisher: Sensory Channels
Kinesthesia
• Perception of limb position, motion, force
• Main information from mechanoreceptors
– “Force sensors”: golgi tendon organsMeasure tension between muscles & tendons
– “Position sensors”: muscle spindlesLocated between individual muscle fibersexcited by changes in muscle length (stretch)
Fisher: Sensory Channels
Fields of interest: Cognitive Science
• Cogsci Society founded 22 years ago• Combines Experimental Psychology, AI,
Philosophy and Neurophysiology • 3 levels of analysis
– Semantics: Intentions, Goals, and Meanings– Syntax: Information processing– Implementation: Neural processing
• Goal is often Cognitive Architecture
Fisher: Sensory Channels
Other fields of interest
• Psycholinguistics
• Social cognition
• Cultural Psychology
• Communication theory
• Embodied communication, paralinguistics
• Anthropology
Fisher: Sensory Channels
Example of metacognitive gap
• The “computer metaphor” of mind
• Intuition: Single mental processor reads all
senses and performs a variety of tasks.
• Cognitive Science: Processing modules
operate in parallel.
– Restricted flow of information and control.
– Processing characteristics are counterintuitive
Fisher: Sensory Channels
“Horizontal” Modularity
• Model Human Processor (MHP)
• Serial stages of processing
• Information flow is Bottom-up
– Cognitive impenetrability
– “Seeing is believing”
Action
Cognition
Perception
Fisher: Sensory Channels
Horizontal modularity restrictions
Cognitive impenetrability (Pylyshyn, 1984) refers to the inability of observers to use semantic information (such as what the person believes or intends to do) to influence the operation of the input stage.
Fisher: Sensory Channels
Vertical modularity (Fodor)
Phoneme perception
Voice recognition
Auditory localization
Cognitive processing
Fisher: Sensory Channels
Vertical modularity restrictions
Information encapsulation (Fodor, 1983) refers to structural barriers within the
cognitive architecture that prevents internal data stores from being shared between
modules in the same stage of processing.
Fisher: Sensory Channels
Events are processed by many channels
• Intuitions about thinking
– Fails at low levels
• Cognitive Architecture
– Multiple brain areas
– Interconnected
– Informationally encapsulated
– Multimodal inputs parsed from scene
Cognition
DorsalSystem
Bimodalspeech
Ventralsystem
…
Sensory world
Fisher: Sensory Channels
Cognitive Impenetrability & Modularity
• Stroop effect– Reading is data-driven module– Competition for response
• Other Modularity phenomena– Modularity of perception for action– Modularity of visual/auditory integration– Modularity of eye movement control– Modularity for Models of Minds
Fisher: Sensory Channels
Advantages of modular processing
• Download task to module, reduce cognitive bottlenecks
• Fast, effortless information processing
• Near-optimal information integration between cues and sensory modalities– Fuzzy logic cue integration
– Bayesian categorization
Fisher: Sensory Channels
"Civilization advances by extending the number of important operations which we can perform without thinking about them."
Alfred North Whitehead
What are the disadvantages?
Fisher: Sensory Channels
Disadvantage: Processing rigidity
• Stress within a module is not accessible• Complex stimuli may be processed
differently in different modules• Tasks may access different modules with
different performance characteristics• Virtual environments may introduce
discrepancies that impact different modules (and tasks) differently
Fisher: Sensory Channels
Stress within a module may be undetected
Poor cognitive access to low-level processes
Action
Cognition
Perception
Fisher: Sensory Channels
Example: VDT stress syndrome
• Users complain of headache, vision problems etc.
• Reports anecdotal, but reading impairment is observed
• Also pupillary tremor and regressive saccades
Fisher: Sensory Channels
Reflective HCI Practice (after Schön)
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Implement
Interaction Design
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fisher: Sensory Channels
Reflective HCI Practice: Foraging
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fisher: Sensory Channels
Theory of space constancy in active vision
• Cognition needs the illusion of a stable world
• Eye movements should create confusing image shifts
• Maintaining space constancy requires– Efference copy
– Passive blur
– Lateral masking
– Saccadic suppression
Fisher: Sensory Channels
Reflective HCI Practice: Hypothesizing
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fisher: Sensory Channels
VDT fatigue study
• Hypothesis “Sampling” raster during saccades reduces intra-saccadic blur, and may overload saccadic suppression
• Space constancy perspective allowed us to:– Isolate the important factors in a complex
situation
– Find a more sensitive task and measure
• Study examines detection of movement during saccade (w Bridgeman & Macnik)
Fisher: Sensory Channels
VDT fatigue study recommendations
• Suppression thresholds elevated for flickering stimuli
• Effect is reduced >250 Hz
• Problems will be greatest – large saccades
– high-contrast display
• Work arounds include blanking display during saccades
Fisher: Sensory Channels
Reflective HCI Practice: Testing
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Implement
Interaction Design
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fisher: Sensory Channels
Moving to multimodality
Vision
Virtualinteraction model
Force & tactilefeedback
Psychophysics of vision, sound, and touch will change when environment is multimodal
Hearing
Fisher: Sensory Channels
Extending to complex worlds
• Lab studies: few events, visual or auditory
• In contrast to multimodal interfaces
– Virtual worlds
– Augmented reality
– Ubiquitous computing
• How are multiple multimodal events dealt
with in the brain?
Fisher: Sensory Channels
Some systems are multimodal
Example: Cross-modal speech system
• Reduces cognitive load
– Fast, effortless information processing
• Near-optimal information integration between cues and sensory modalities
– Fuzzy logic cue integration
– Bayesian categorization
Fisher: Sensory Channels
Cross-modal speech perception
The McGurk Effect: McGurk & MacDonald 1976
Fisher: Sensory Channels
Illusory conjunctions occur in artificial multimodal environments
Example: Movie theatre
• The McGurk effect (face influences sound)
– Dubbed movie
• The ventriloquist effect (vision captures sound location)
– Sound seems to come from actor
Fisher: Sensory Channels
Rensink
Fisher: Sensory Channels
Study: Pointing to sounds • Cognitive location good
• Pointing shows visual capture
– Aware of visual and auditory locations, but point to visual
• No effect of phoneme/viseme fit on ventriloquism
• Slow recalibration of auditory space if offset constant
“Ba”
What was sound?Where was source?(point or describe)
Fisher: Sensory Channels
Our interpretation: 2 systems at work
• Different multimodal systems solve feature
assignment problem differently
– Motor system: high visual dominance
– Cognitive system: low visual dominance
– Phoneme/viseme mismatch doesn’t help
• Vision can recalibrate spatial sound “map”
Fisher: Sensory Channels
Rensink
Fisher: Sensory Channels
Attentional pointers in systems?
Cognitive processing
Action (motor space)
Auditory localization
Visual localization
Fisher: Sensory Channels
Study: Pointing in large displays (Po)
• Tell me where the target is
• Point with no feedback
• Point with visual feedback (cursor)
• Point with delayed visual feedback
Fisher: Sensory Channels
Findings
1. Can you tell if a target is on the left or right?
• 3 out of 7 males, 7 out of 7 females made errors
2. Can you point to it with no visual feedback?
• 6 out of 10 who failed #1 were correct
3. Are you better with a (simulated) laser pointer?
• Out of 6 who point accurately in 2, all fail
4. Will pointing accuracy be affected if visible pointer lags
pointing?
• 3 of the 6 who failed #3 succeed
All results predicted by 2 visual systems
hypothesis
Fisher: Sensory Channels
Displays and multimodal perception
• Immersive environments must display a complex multimodal world– Virtual Reality must provide entire world– Augmented Reality must blend with real world
• Multimodal displays have errors– Location of events is not precise (esp. in depth)– Timing is not precise– Graphics can be low-fidelity
• What will be the impact of these errors on users?
Fisher: Sensory Channels
Disadvantage: Each module must solve feature assignment problem.
• Modules can’t accept information from other modules: Information encapsulation
• Different modules should have access to a different set of matching cues.
• Illusory conjunctions can occur in multimodal environments:– Phoneme perception: The McGurk effect
– Auditory localization: The ventriloquist effect
Fisher: Sensory Channels
Study: Impact of display errors on multimodal perception
• Immersive environments typically have display errors– Location of events is not precise– Timing is not precise– Graphics can be low-fidelity
• As immersive environments add sound and touch, what will be the impact of these errors?
Fisher: Sensory Channels
Reflective HCI Practice:Foraging
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fisher: Sensory Channels
Recalibration by pairing (Epstein, 1975)
• Individual senses adapt to display • Sensory modalities calibrate each other:
haptics, vision, sound– Observed actions calibrate visual space (space
constancy) – Vision calibrates hearing for the location of a
multimodal event– Sound calibrates vision for the time of a
multimodal event
• Result is an after-effect: remapping of auditory (visual, haptic) space
Fisher: Sensory Channels
Reflective HCI Practice: Hypothesizing
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fisher: Sensory Channels
Impact of information encapsulation
• Multimodal environment with errors in timing and location
• The same event might give rise to a single multimodal construct in one task, and two unimodal events for another.– Vary location of visual and auditory phonemes
in a simple teleconferencing-style video display
– Vary information carried by using synthetic speech stimuli (5 levels).
Fisher: Sensory Channels
Ventriloquism meets the McGurk effect.
• Vary location of visual and auditory phonemes in a simple teleconferencing-style video display
• Vary information carried by using synthetic speech stimuli (5 levels).
• Subjects report sound location and syllable heard,
• Analyses included testing a variety of mathematical models of information integration by fitting free parameters with STEPIT.
Fisher: Sensory Channels
Use of mathematical modeling tools allow us to address
• Sensory input from a number of channels simultaneously
• How stimuli from multiple channels are matched and partitioned into mental representations
• How information from multiple senses is integrated to give rise to trans-modal mental events
Fisher: Sensory Channels
Reflective HCI Practice: Testing
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Implement
Interaction Design
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fisher: Sensory Channels
Results:
• Visual capture of auditory source location, resulting in a shifting of unimodal auditory location estimation (ventriloquism after-effect).
• No effect of location difference on phoneme perception as measured by statistical or modeling tests.
• No correlation between errors in the two tasks (i.e. subjects could not selectively attend to the auditory phoneme on trials when visual capture failed).
• Overall, modularity of phoneme perception is supported.
Fisher: Sensory Channels
Disadvantage: Changes in task interact with modules to change performance
• 2 visual systems—“ventral stream” for recognition and “dorsal stream” for action.
• Where vs how• Different impact of illusions• Lesion data
Fisher: Sensory Channels
Functional Neuroanatomy of perception for action.
2 visual systems—“ventral stream” for cognition and “dorsal stream” for motor performance.
Fisher: Sensory Channels
2 visual systems lesion evidence
lesion performance deficits spared abilities
V1 (blindsight) detection and identification
pointing
Ventrolateral occipital (DF)
identification, shape recognition, object orientation
object manipulation (orientation matching, grip scaling)
Posterior parietal (RV)
object manipulation (orientation matching, grip scaling)
identification, shape recognition, object orientation
Fisher: Sensory Channels
2 visual system illusions
stimuli deficits spared abilitiesTichner circles size report grip scaling
displacement during saccade
detection of displacement, location report
pointing
Moving or off-centre frame
induced motion, location report
pointing
Fisher: Sensory Channels
Research with videoconferencing and abstract displays
• Targeting sound: cognitive better than motor – Subs aware of visual and auditory
locations, but point to visual
• Targeting vision with context: Less feedback is better– Pointing with no visual feedback better
– Lagged cursor better than unlagged
Fisher: Sensory Channels
Interpreting pointing studies
• Pointing studies counterintuitive, but
predicted by response characteristics
of neurons in dorsal/ventral to visual
and auditory stimuli
Fisher: Sensory Channels
2 visual system illusions
stimuli deficits spared abilitiesTichner circles size report grip scaling
displacement during saccade
detection of displacement, location report
pointing
Moving or off-centre frame
induced motion, location report
pointing
Sound with displaced visual distractor
pointing apparent location of sound
Fisher: Sensory Channels
Modularity Recap
• Displays can interact with active perception to cause hidden stress: VDT study
• Displays can impact matching cues for multimodal cue integration– Support or frustrate recalibration by pairing– Lead to discrepancies in number and
composition if stimuli in different modules• Displays can cause disagreement between
motor performance and cognitive measures• Sometimes removing information helps
Fisher: Sensory Channels
Extending to complex worlds
• Previous studies in simple worlds, with a few visual and auditory events
• Multimodal environments are complex– Virtual worlds
– Augmented reality
– Ubiquitous computing
• How are multiple multimodal events dealt with in the cognitive architecture?
Fisher: Sensory Channels
Indexical cognition (Pylyshyn)
Fisher: Sensory Channels
“FINSTs... make thoughts true”
• Perception– “Hotlink” tokens– Drawn to salient events– Object-centred, “sticky”– Visual routines– Finite number ~ 4
• Cognition– Maintain object history– Implicit memory of object associations– Sparse cognitive representation– Just-in-time delivery of information– Atom of intentionality
Fisher: Sensory Channels
Mental representations of complex worlds
• Cognitive architecture perspective requires that links be established between lower level perceptual qualities and cognitive symbols—i.e. a pointer, called a FINST.
• FINSTing allows us to interact with perceptual objects and events without the need for mental images per se.
• Symbolic representation + pointers makes different predictions than intuitive picture-in-the-head
Fisher: Sensory Channels
Indexical cognition (Pylyshyn)
Fisher: Sensory Channels
Naïve view of FINSTs in Cognitive Arch
Phoneme perception
Voice recognition
Auditory localization
Cognitive processing
Action (motor space)
FINSTs
Fisher: Sensory Channels
Another view of FINSTs
Phoneme perception
Voice recognition
Auditory localization
Cognitive processing
Action (motor space)
FINSTs
Fisher: Sensory Channels
More about FINSTs
• FINSTs Link mind & perceptual world– Visual routines: (collinear, inside,
subitizing)– History of an object– Object-centred, “sticky”– Drawn to salient changes-- onsets,
luminance increments, oddballs– Finite number ~ 4-7– FINSTs + ANCHORs for motor
behaviour
Fisher: Sensory Channels
More about ANCHORs
• ANCHORs link mind & action– Remembered locations for eye
movements
– Direct interaction with items off the retina
– Fast, robust motor performance by action routines
– Affordances for action
Fisher: Sensory Channels
Role of haptics
• See “physical interaction:human haptics” by Karon MacLean
Fisher: Sensory Channels
Multiple object tracking demo
Fisher: Sensory Channels
Another trial (Scholl)
Fisher: Sensory Channels
Another trial (Scholl)
Fisher: Sensory Channels
Application
Fisher: Sensory Channels
Mental representations of complex environments
• Cognitive architecture perspective requires that links be established between lower level perceptual qualities and cognitive symbols—i.e. a pointer, called a FINST.
• FINSTing allows us to interact with perceptual objects and events without the need for mental images per se.
• Symbolic representation + pointers makes different predictions than intuitive picture-in-the-head
• Coping with spatial transformations in complex data spaces
Fisher: Sensory Channels
Multimodal representations are virtual
All modalities store little info in memory:
instead they take up information as needed
– Vision-- attention, eye, head and body
movements change view
– Haptics-- active exploration of space with hands
– Hearing-- uses body and head movements to
localize sound and improve quality
Fisher: Sensory Channels
Multimodal events support adaptation
• Individual senses adapt to display
• Modalities use multimodal events for cross-calibration– Observed actions calibrate visual space
– Vision calibrates sound location
– Sound calibrates vision for time
• Result includes after-effect: a remapping of perceptual space
(Epstein, 1975)
Fisher: Sensory Channels
Research question: Role of focal attention?
Are attentional resources shared between senses?
• Will adding sound and haptics impact visual attention?– Or, will it offload processing from vision?
• Does a shift in one modality cause complementary attention shifts in others?
• Does recalibration require attention?
Fisher: Sensory Channels
Research Topic: Pointers for action?
• Attentional pointers link mind and world
• Do “action pointers” link mind & muscles?– Remembered locations for eye movements
– Direct interaction with items off the retina
– Fast, robust motor performance by action routines
Fisher: Sensory Channels
Research Topic: Individual differences
• Perceptual rules are the same
• Impact differ over time and for individuals
– e.g. sensitivity to stereo depth & spatial sound cues
– Ability to adapt to new cue combinations
• Perceptual customization may help
– For individuals: “personal equation” for interaction
– In real time, through attentive computing
Fisher: Sensory Channels
Module advantages
• High-realism interface designs improve
performance by “downloading” information
processing operations to input modules.
• Interaction of display characteristics with
capabilities and characteristics of the
functional architecture will determine
performance.
Fisher: Sensory Channels
Module disadvantages
• Coordination– Distortions in location, timing, and category-relevant
information may lead to the formation of conflicting representations in different modules.
• Processing inflexibility– Errors and conflicts within a module can create errors and
increase cognitive load. (CRT flicker example)
• Information hidingCognitive impenetrability of modules makes it difficult for
operators to determine the reasons for their poor performance.
Fisher: Sensory Channels
Inseparability of Mind & World
• Embodied cognition-- mind/body
• Situated cognition-- mind/world
• Distributed cognition-- mind/mind
• Ecological theories (Vygotski, Luria, Bateson, Gibson) can be linked to sensory phenomena and inform interaction design
Fisher: Sensory Channels
Key Points for HCI Practice
• The Metacognitive Gap: The need for a grounded Cognitive Science approach
• Reflective design practice methods– Integrating CogSci with interaction design– Iterative design cycle
• Examples of extended HCI – Cognitive architecture: Multimodal displays and
how they are understood – Situated cognition: embodied interaction with
complex displays
Fisher: Sensory Channels
Environment/Inhabitant/Representation
• Information visualization
• Perceptual/deictic/situated Cognition
• Cognitive Architecture, perception, attention
• Display as extension of mental model
Fisher: Sensory Channels
What to expect in the next talk
• More on haptics• Other senses
– Neuromuscular,GSR, heart rate, brain, other biopotentials
• Applications– Displays, input, & sensing technologies– Design examples– Virtual environments
• Communicating human experience: information, emotion, environment
– Intimacy and embodiment– Sources of aesthetics