ICS3211 - Intelligent

Interfaces IICombining design with technology for effective

human-computer interaction

Week 7Department of Intelligent Computer

Systems,University of Malta,

2016

Design for Multimodal Interfaces

Week 6 overview:

• Paper Prototype - Task 3 (1 hour)

• Multimodal interactions

• Human interaction in multimodal systems

• Design guidelines

• Real world systems

Learning OutcomesAt the end of this session you should be able to:

• Describe the characteristics of multimodal interfaces;

• Draw inferences about the design of multimodal interfaces;

• Compare and contrast the multiple modalities which interfaces would require depending on the context;

• List the best practices of the design principles for multimodal interfaces;

Paper Prototyping - Task 3• Set up your project around the lab;

• 1 person from the team facilitates the prototyping exercise;

• 1-2 persons will move around the available project prototypes;

• Prototype facilitator briefs user, describes main task, records observations, photos/videos, etc.

Multimedia vs. Multimodal

• Multimedia – more than one mode of communication is output to the user; e.g. a sound clip attached to a presentation.• Media channels: text, graphics, animation,

video: all visual media• Multimodal – computer processes more than

one mode of communication; e.g. the combined input of speech and touch in smart phones• Sensory modalities: Visual, auditory, tactile,

Multimodal Interactions• Traditional WIMP offers limited input/output

possibilities;• Mix of audio/visual interactions important for

communication;• All senses (including touch) are relevant;• Combination of multiple modalities (including

speech, gestures, etc.) offer new functionalities.

Multimodal Interactions• Modality is the mode or path of communication

according to human senses, using different types of information and different interface devices;

• Some definitions:

Multimodal HCI system is simply one that responds to inputs in more than one modality or communication channel (e.g. speech, gesture, writing and others) [James/Sebe]

Multimodal interfaces process two or more combined user input modes (such as speech, pen, touch, manual gesture, gaze and head and body movements) in a coordinated manner with multimedia system output. [Oviatt]

Multimodal Interactions

• Use this padlet [https://padlet.com/vanessa_camille/multimodal_ICS3211 ] to list how the two modalities using speech and gesture differ;

Input Modalities• Speech or other

sounds• Head movements

(facial expression, gaze)

• Pointing, pen, touch• Body

movement/gestures • Motion controller

(accelerometer)• Tangibles• Positioning• Brain-computer

interface• Biomodalities

(sweat, pulse, respiration)

Output Modalities• Visual:

• Visualization• 3D GUIs• Virtual/Augmented

Reality• Auditory:

• Speech – Embodied Conversational

• Sound• Haptics

• Force feedback• Low freq. bass• Pain

• Taste• Scent

Speech vs. Gestures• Information that can be accessed from

speech: • Word recognition• Language recognition• Speaker recognition• Emotion recognition• Accent recognition

Speech vs. Gestures• Humans use their body as communication

modality: • Gestures (explicit & implicit)• Body language• Focus of attention• Activity

• Perception by computers: • Computer vision• Body mounted sensors

Haptics

• Manipulation tasks require feeling of objects;• Computers can perceive this by:

• Haptic interfaces • Tangible objects• Force sensors

Biophysiological Modalities

• Body information through: • Brain activity• Skin conductance• Temperature • Heart rate

• Reveal information on: • Workload• Emotional state• Mood• Fatigue

Types of Multimodal Interfaces

• Perceptual• highly interactive• rich, natural interaction

• Attentive• context aware• implicit

• Enactive• relies on active manipulation through the use of hands or

body, such as TUI

Challenges of Multimodal Interfaces

• Development of cognitive theories to guide multimodal system design

• Development of effective natural language processing

• Dialogue processing• Error-handling techniques• Function robustly and adaptively• Support for collaborative multi-person use

Design of Multimodal Interfaces

• Multimodal interfaces are designed for:• compatibility with users’ work practices;• flexibility;

• Design criteria;• robustness increases as the number and

heterogeneity of modalities increase;• performance improves with adaptivity of interface;• persistence of operation despite physical damage,

loss of power, etc.

Guidelines for the Design of Multimodal Interfaces

• To achieve more natural interaction, like human-human interaction

• To increase robustness by providing redundant and complementary information

Guidelines for the Design of Multimodal Interfaces

1.Requirements specifications• design for broad range of users (experience, abilities, etc.) and

contexts (home, office, changing environments like car)• address privacy and security issues• don’t remember users by default• use non-speech input for private information, like passwords

2.Designing multimodal input and output• guidelines stem from cognitive science:

• maximize human cognitive and physical abilities e.g., don’t require paying attention to two things at once

• reduce memory load

• multiple modes should complement each other, enhance each other

• integrate modalities to be compatible with user preferences, context and system functionality e.g., match input and output styles

• use multimodal cues, e.g., look at speaker• synchronize modalities (timing)• synchronize system state across modalities

3. Adaptivity• adapt to needs/experiences/skill levels of different users and

contexts• examples: gestures replace sounds in noisy settings,

accommodate for slow bandwidth, adapt quantity and stye of information display based on user’s perceived skill level

4.Consistency

• use same language/keywords for all modalities

• use same interaction shortcuts for all modalities

• support both user and system switching between modalities

5.Feedback

• users should know what the current modality is and what other modalities are available

• avoid lengthy instructions

• use common icons, simple instructions and labels

• confirm system interpretation of user’s commands, after fusion of all input modalities has completed

6.Error preventing and handling

• clearly mark “exits” from: task, modality & system

• support “undo” & include help

• integrate complementary modalities to improve robustness: strengths of one modality should overcome weaknesses of others

• let users control modality selection

• use rich modalities that can convey semantic information beyond simple

• point-and-click

• fuse information from multiple sources

• Users do like to interact multimodally with artificial systems

• Multimodal interaction is preferred in spatial domains; we should offer & expect more than point & speak

• Multimodal interaction changes the way we talk; we need to adapt our speech processing components

• Speech and gesture are synchronized but not simultaneous;

• We cannot assume redundancy of content; we need to process modalities in an integrated manner

• Use multimodality for better system error characteristics, through expecting simplified speech, fusion of modalities to clear out uncertainty & offering the right modality for the right task

Myths of Multimodal Interfaces1.If you build a multimodal system, users will interact multimodally

2.Speech and pointing is the dominant multimodal integration pattern3.Multimodal input involves simultaneous signals4.Speech is the primary input mode in any multimodal system that

includes it5.Multimodal language does not differ linguistically from unimodal

language6.Multimodal integration involves redundancy of content between

modes7.Individual error-prone recognition technologies combine multimodally

to produce even greater unreliability8.All users’ multimodal commands are integrated in a uniform way9.Different input modes are capable of transmitting comparable content10.Enhanced efficiency is the main advantage of multimodal systems

Case ExampleBohus, D., & Horvitz, E. (2010, November). Facilitating multiparty dialog with gaze, gesture, and speech. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (p. 5). ACM.

System Architecture

Behaviour Patterns

Key Concepts in Multimodal Interfaces

• We cannot always believe our intuition on how interaction will function; we need to find out by performing well designed user studies

• We need to take a close look at how human – human communication and interaction works before we can build systems that resemble this behaviour; costly collection and annotation of real world data is necessary

• We need to generate semantic representations from our sub-symbolic feature space