Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g....

30
Multimodal Interaction

  • date post

    20-Dec-2015

  • Category

    Documents

  • view

    223

  • download

    0

TAGS:

Transcript of Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g....

Page 1: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Multimodal Interaction

Page 2: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Modalities vs Media

Modalities are ways of encoding informatione.g. graphics

Media are instantiations of modalitiese.g. a particular image

Page 3: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

How Do Multimodal Systems Differ?

Domain/application Available media Modeling of context/environment Modeling of user Focus of research

Page 4: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Example Multimodal Systems

Not speech-centricMIT paintbrush, soundbrush

• http://www.youtube.com/watch?v=04v_v1gnyO8

• http://www.youtube.com/watch?v=iZbe3t8YSf4

• http://www.youtube.com/watch?v=18RY8Jgid20

Wearables• http://www.gatech.edu/innovations/wearable/

Page 5: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Example Multimodal Systems

Speech-centricMSOIPCOMICSmartKom

Page 6: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

MSOIP Keywords

Multimodal mobile dialog Integration of speech and pen input User modeling for presentations

Johnston et al. 2001

Page 7: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

MATCH Video

http://www.research.att.com/~johnston/Scroll down to the bottom of the page

Page 8: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

About MATCH

What input modalities? What output modalities? What application(s)? What aspects of context?

Page 9: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

COMIC Keywords

Ambient intelligence HHI/HCI research

Collaborative problem solving User modeling Avatar

Alexandersson et al. 2004

Page 10: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

COMIC Video

http://www.hcrc.ed.ac.uk/comic/demos/facial-animation/

Page 11: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

COMIC Video

http://www.hcrc.ed.ac.uk/comic/demos/slot/

Page 12: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

About COMIC

What input modalities? What output modalities? What applications? What aspects of context?

Page 13: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

SmartKom Keywords

Multimodal dialog across applications devices and situations

Avatar Situation aware

Alexandersson et al., Reithinger et al. 2003

Page 14: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

SmartKom Video

http://www.smartkom.org/start_en.htmlI showed the SK-Mobile one, but theother one is also interesting.

Page 15: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

About SmartKom

What input modalities? What output modalities? What applications? What aspects of context?

Page 16: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Parts of a Multimodal System

Interpreter

DialogManager

Generator

KnowledgeBase

SpeechIn

SpeechOut

GestureIn

PresentOut

TextIn

TextOut

Page 17: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

HCI and Multimodal Systems

Input integration/fusion Representations Effective help Quality presentations Managing context Understanding the user

Page 18: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Different Uses of Modalities

Concurrent or sequential

Redundant or Complementary or Contradicting

Page 19: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Input Integration/Fusion

Key elements:TimeMultiple uses of some modalitiesError rates

Typical approach is to map straight to semantics if possible

Page 20: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Representation

Increasing use of XML-based languages (SMIL, EMMA)But these don’t solve the semantic

problems Keep ‘backbone’ knowledge separate

from ‘peripheral’ information (Alexandersson et al.)

Page 21: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Effective Help

How do each of the systems provide the user with:Explicit help?Implicit help?

Page 22: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Quality Presentations

Talking headsAdvantagesDisadvantages

Informative presentations are key User modeling/adaptive presentations

are a bonus These systems go beyond scripts

Page 23: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Managing Context

What kinds of context are there in a mobile multimodal interaction?

Page 24: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Understanding the User

What kinds of information can we gather about users in general?

About one user in particular? How can we use this information?

Page 25: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Commercial Multimodal Systems Most are for research

Military• Training and battlefield

Education• Tutoring systems

Commercial ones include: Wii: http://www.youtube.com/watch?

v=n4nZVAEeitU Microsoft surface:

http://www.youtube.com/watch?v=rP5y7yp06n0

Page 26: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

TradeOffs

You get:More intuitive technologyMore information, more easilyLess (dumb stuff) for you to do

You trade:PrivacyControl

Page 27: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Towards the Future

DesignMultimodal systems in virtual worlds,

or crossing over from virtual to real worlds

Ambient multimodal interaction Implementation

Mashups – user controlledPervasive multimedia

Page 28: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Towards the Future

http://www.youtube.com/watch?v=FMJwURqpFWs

http://www.programmableweb.com/mashups

Page 29: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

SciFi?

Lathe of Heaven by Ursula LeGuin Summa Technologiae by Stanislaw Lem Fast Times at Fairmont High by Vernor

Vinge The Human Machine Merger, talk by

Raymond Kurzweil (at http://www.kurzweilai.net/meme/frame.html?main=memelist.html?m=6%23581)

Page 30: Multimodal Interaction. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular.

Additional Info

http://search.techrepublic.com.com/search/multimodal+system.html

http://www.w3.org/2002/mmi/


Therapeutic Modalities Review

Therapeutic Modalities Review

Light Modalities

Light Modalities

Eclipse for Smalltalkers - Instantiations · Eclipse Platform Technical Overview, OTI 2001. Eric Clayberg - Instantiations, Inc. July 14, 2003 Eclipse for Smalltalkers - 5 What is

Eclipse for Smalltalkers - Instantiations · Eclipse Platform Technical Overview, OTI 2001. Eric Clayberg - Instantiations, Inc. July 14, 2003 Eclipse for Smalltalkers - 5 What is

SC1 Workshop 2 Pilot instantiations

SC1 Workshop 2 Pilot instantiations

Functional neuroanatomy of gesture–speech integration in ...€¦ · gist of the story (e.g., flapping palms + “pet”) than when no integration across modalities is required

Functional neuroanatomy of gesture–speech integration in ...€¦ · gist of the story (e.g., flapping palms + “pet”) than when no integration across modalities is required

SURGICAL MODALITIES

SURGICAL MODALITIES

Modalities SFURTI

Modalities SFURTI

Electrical Modalities

Electrical Modalities

Modalities in Rehab · Modalities in Rehab Faizan zaffar kashoo most commonly used cryotherapy thermotherapy electrical stimulation iontophoresis traction ultrasound Modalities in

Modalities in Rehab · Modalities in Rehab Faizan zaffar kashoo most commonly used cryotherapy thermotherapy electrical stimulation iontophoresis traction ultrasound Modalities in

Thermal Modalities ESAT 3640 Therapeutic Modalities.

Thermal Modalities ESAT 3640 Therapeutic Modalities.

UNIT 2: Therapeutic Modalities THERMAL AGENTS 2: therapeutic modalities thermal agents bellwork define the following cryotherapy heat therapy therapeutic modalities

UNIT 2: Therapeutic Modalities THERMAL AGENTS 2: therapeutic modalities thermal agents bellwork define the following cryotherapy heat therapy therapeutic modalities

Medieval Modalities

Medieval Modalities

Modalities in HoTTHigher toposes Internal logic Modalities Sub-1-toposes Formalization Modalities in HoTT Egbert Rijke, Mike Shulman, Bas Spitters 1706.07526

Modalities in HoTTHigher toposes Internal logic Modalities Sub-1-toposes Formalization Modalities in HoTT Egbert Rijke, Mike Shulman, Bas Spitters 1706.07526

A Schema for Constraint Relaxation with Instantiations …eil.utoronto.ca/wp-content/uploads/aac/papers/beck-msc.pdf · A Schema for Constraint Relaxation with Instantiations for

A Schema for Constraint Relaxation with Instantiations …eil.utoronto.ca/wp-content/uploads/aac/papers/beck-msc.pdf · A Schema for Constraint Relaxation with Instantiations for

VA Smalltalk Update - Instantiations · VA Smalltalk Update John O’Keefe Chief Technical Officer Instantiations, Inc. ... • abt.cnf processing has remained in the product as a

VA Smalltalk Update - Instantiations · VA Smalltalk Update John O’Keefe Chief Technical Officer Instantiations, Inc. ... • abt.cnf processing has remained in the product as a

o2 modalities

o2 modalities

INFLUENCE OF RECOVERY MODALITIES ON TEAM SPORT … · In an attempt to optimise recovery, athletes can chose from numerous established recovery modalities (e.g. active and passive

INFLUENCE OF RECOVERY MODALITIES ON TEAM SPORT … · In an attempt to optimise recovery, athletes can chose from numerous established recovery modalities (e.g. active and passive

Therapeutic Modalities 1. THERAPEUTIC MODALITIES 2.

Therapeutic Modalities 1. THERAPEUTIC MODALITIES 2.

Therapeutic Modalities Modalities-treatment of injuries, including heating, cooling, and mechanical/electrical methods. What modalities do we use in our.

Therapeutic Modalities Modalities-treatment of injuries, including heating, cooling, and mechanical/electrical methods. What modalities do we use in our.

Tool Evaluation: EclipsePro Audit by Instantiations · PDF fileTool Evaluation: EclipsePro Audit by Instantiations Teams Diversity + Team13 ... is a J2EE web application which ...

Tool Evaluation: EclipsePro Audit by Instantiations · PDF fileTool Evaluation: EclipsePro Audit by Instantiations Teams Diversity + Team13 ... is a J2EE web application which ...