Nov 19, 2008 1

Deutsche Telekom Laboratories

The W3C Multimodal Architecture

ETSI Workshop: Multimodal Interaction on Mobile Devices, Ingmar Kliche (Nov 19, 2008)

W3C Multimodal Architecture.

Agenda.

� The W3C

� W3C Multimodal Interaction Working Group

� W3C Multimodal Architecture (overview)

� Upcoming web standards

� SCXML (State Chart XML)

Deutsche Telekom Laboratories Nov 19, 2008 2

� SCXML (State Chart XML)

� VoiceXML 3.0

� EMMA (Extensible Multimodal Annotation Markup Language)

� Context awareness in multimodal applications

� A practical implementation of the W3C Multimodal Architecture

� Conclusion

The W3C.

W3C Multimodal Architecture.

The W3C – Mission and Goals.

� Mission: To lead the World Wide Web to its full potential by developing protocols

and guidelines that ensure long-term growth for the Web.

� The main goal: To ensure interoperabilityyyy of web standards, irrespective of

special interests.

Deutsche Telekom Laboratories Nov 19, 2008 4

W3C Multimodal Architecture.

The W3C – Governance and Statistics.

� Web standards creation

� Since 1994 some 110 Web standards have been published.

� Infrastructure

� International representation (ERCIM*, MIT**, Keio Uni.***) with

� 400 member organizations from 40 countries

W3C Hosts and offices map

Deutsche Telekom Laboratories Nov 19, 2008 5

� 400 member organizations from 40 countries

� Accountability

� Vendor-neutral forum

� Public discussion with full consideration of public feedback

� Participation in working groups, as member or invited expert

� Patent policy

� Strong patent policy to protect company and public interests* European Research Consortium for Informatics and Mathematics (ERCIM)

** MIT Computer Science and Artificial Intelligence Laboratory (CSAIL)

*** Keio University

W3C MMI Working Group.

W3C Multimodal Architecture.

Upcoming web standards.Upcoming web standards.

W3C Multimodal Architecture.

W3C Multimodal Interaction Working Group.

� Mission: “… to extend the Web to allow

users to dynamically select the most appropriate

mode of interaction for their current needs ...“

� Participants

Deutsche Telekom Laboratories Nov 19, 2008 7

� 40 participants from 22 organizations

� More information: http://www.w3.org/2002/mmi

W3C Multimodal Architecture.

Multimodal Interaction Framework.

� W3C Multimodal Interaction Framework (published 2003)

Deutsche Telekom Laboratories Nov 19, 2008 8

W3C Multimodal Architecture.

“Multimodal Architecture and Interfaces”.

“Multimodal Architecture and

Interfaces“ (working draft status)

� Runtime Framework provides the basic

infrastructure and controls

communication among the

constituents. Runtime Framework

Data Component

InteractionManager

DeliveryContext

Component

Deutsche Telekom Laboratories Nov 19, 2008 9

constituents.

� Interaction Manager (IM) coordinates

modality components (MCs) by life-

cycle events and contains the shared

data (context).

� Event-based communication between

IM and MCs.

Modality Component 1

Modality Component N

Modality Component API

W3C Multimodal Architecture.

“Multimodal Architecture and Interfaces”.

“Multimodal Architecture and

Interfaces“ (working draft status)

� MMI lifecycle events

� newContextRequest/Response

� startRequest/Response Runtime Framework

Data Component

InteractionManager

DeliveryContext

Component

Deutsche Telekom Laboratories Nov 19, 2008 10

startRequest/Response

� cancelRequest/Response

� prepareRequest/Response

� statusRequest/Response

� pauseRequest/Response

� resumeRequest/Response

� Done

� extension

Modality Component 1

Modality Component N

Modality Component API

<scxml initial= "foo" ...><datamodel>

<data id= "x" expr= “’1’"/ ></datamodel>

<scxml initial= "foo" ...><datamodel>

<data id= "x" expr= “’1’"/ ></datamodel>

<scxml initial= "foo" ...><datamodel>

<data id= "x" expr= “’1’"/ ></datamodel>

<scxml initial= "foo" ...><datamodel>

<data id= "x" expr= “’1’"/ ></datamodel>

W3C Multimodal Architecture.

State Chart XML (SCXML).

“State Chart XML (SCXML): State Machine Notation for Control Abstraction”

(working draft status)

Deutsche Telekom Laboratories Nov 19, 2008 11

</datamodel>

</scxml>

</datamodel><state id= "foo" >

</state><state id= "bar" >...</state>

</scxml>

</datamodel><state id= "foo" >

<transition event= "foo"target= "bar" >

</transition></state><state id= "bar" >...</state>

</scxml>

</datamodel><state id= "foo" >

<transition event= "foo"target= "bar" >

<send event= " play " target= " VUI "targettype= " basichttp " />

</transition></state><state id= "bar" >...</state>

</scxml>

W3C Multimodal Architecture.

Voice Modality Component (VoiceXML).

W3C VoiceBrowser Working Group

� Existing standards:

� VoiceXML 2.1

� CCXML 1.0

� SRGS 1.0 and SISR 1.0

PLS 1.0 and SSML 1.0

<vxml version=“3.0”xmlns=“...” ...>

<form id= "foo" ><field name=" foo " >

Deutsche Telekom Laboratories Nov 19, 2008 12

� PLS 1.0 and SSML 1.0

� Work in progress:

� SCXML 1.0

� VoiceXML 3.0

� External eventing

� New features (e.g. speaker verification)

� Alignment with MMI architecture

<field name=" foo " ><grammar .../><prompt .../>

</field><filled>

<send event=.../></filled>

</form>

<form id= "bar" >...</form>

</vxml>

* Note: VoiceXML 3 syntax not yet finalized, example shows principle.

<emma:emma version="1.0“ xmlns:emma=“ http://www.w3.org/2003/04/emma ” …><emma:emma version="1.0“ xmlns:emma=“ http://www.w3.org/2003/04/emma ” …><emma:emma version="1.0“ xmlns:emma=“ http://www.w3.org/2003/04/emma ” …><emma:emma version="1.0“ xmlns:emma=“ http://www.w3.org/2003/04/emma ” …>

“EMMA: Extensible MultiModal Annotation markup language”

� EMMA 1.0 (www.w3.org/TR/emma) currently in Candidate Rec. status

� Goal: Annotation/representation of user input

� Example: user utterance “flights from Boston to Denver“

W3C Multimodal Architecture.

EMMA.

Deutsche Telekom Laboratories Nov 19, 2008 13

<emma:emma version="1.0“ xmlns:emma=“ http://www.w3.org/2003/04/emma ” …>

</emma:emma>

<emma:emma version="1.0“ xmlns:emma=“ http://www.w3.org/2003/04/emma ” …>

<emma:interpretation id="int1" emma:confidence="0.75“emma:tokens="flights from boston to denver">

<origin> Boston </origin><destination> Denver </destination>

</emma:interpretation>

</emma:emma>

<emma:emma version="1.0“ xmlns:emma=“ http://www.w3.org/2003/04/emma ” …><emma:one-of id="r1" emma:start="1087995961542" emma:end="1087995963542“

emma:medium="acoustic" emma:mode="voice"><emma:interpretation id="int1" emma:confidence="0.75“emma:tokens="flights from boston to denver">

<origin>Boston</origin><destination>Denver</destination>

</emma:interpretation>

</emma:one-of></emma:emma>

<emma:emma version="1.0“ xmlns:emma=“ http://www.w3.org/2003/04/emma ” …><emma:one-of id="r1" emma:start="1087995961542" emma:end="1087995963542“

emma:medium="acoustic" emma:mode="voice"><emma:interpretation id="int1" emma:confidence="0.75“emma:tokens="flights from boston to denver">

<origin> Boston </origin><destination> Denver </destination>

</emma:interpretation><emma:interpretation id="int2" emma:confidence="0.68“emma:tokens="flights from austin to denver">

<origin> Austin </origin><destination> Denver </destination>

</emma:interpretation></emma:one-of>

</emma:emma>

W3C Multimodal Architecture.

The Markup View.

� Implementation of the W3C Multimodal Architecture using W3C markup

languages.

Data Component

InteractionManager

DeliveryContext

ComponentSCXML

Deutsche Telekom Laboratories Nov 19, 2008 14

Modality Component 1

Modality Component N

Runtime Framework

Modality Component API Modality Component API

HTMLfor GUI

SCXML

VoiceXMLfor VUI

EMMA

to be use within MMI

lifecycle events to represent

user input

Context Awareness in multimodal

applications.

W3C Multimodal Architecture.

Related W3C Standards.

� The W3C has developed other standards which are interesting in the context of

multimodal applications. e.g.:

� DCCI (Delivery Context Client Interfaces)

� http://www.w3.org/TR/DPF

Deutsche Telekom Laboratories Nov 19, 2008 16

� http://www.w3.org/TR/DPF

� Accessing Static and Dynamic Delivery Context Properties

� Device Description Repository

� http://www.w3.org/TR/DDR-Simple-API

� Simple API to access device information

W3C Multimodal Architecture.

Delivery Context Information.

� Delivery Context Client Interfaces (DCCI): Client interface to access (static and

dynamic) device information, e.g. loudspeaker volume

Data Component

InteractionManager

DeliveryContext

ComponentSCXML

Data model

Deutsche Telekom Laboratories Nov 19, 2008 17

Modality Component 1

Modality Component N

Runtime Framework

Modality Component API Modality Component API

HTMLfor GUI

SCXML

VoiceXMLfor VUI

Data model

DCCI

Related activities with regards to

Multimodal Interaction.

W3C Multimodal Architecture.

Upcoming Activities.

� Emotion Incubator Group:

� Annotation of Emotions (http://www.w3.org/2005/incubator/emotion)

� Goal: Standardization of Emotion Markup Language (EmotionML)

� Model based User Interfaces Incubator Group:

Deutsche Telekom Laboratories Nov 19, 2008 19

� Model based User Interfaces Incubator Group:

� Evaluation of research on model-based user interface design

(http://www.w3.org/2005/incubator/model-based-ui)

� Source of further information: UsiXML (http://www.usixml.org)

W3C Multimodal Architecture.

Upcoming Activities.

� Geolocation Working Group:

� Provisioning of location information within web browser environments

(http://www.w3.org/2008/geolocation)

� Rich Web Application Backplane Incubator Group:

Deutsche Telekom Laboratories Nov 19, 2008 20

� Rich application backplane

(http://www.w3.org/2005/incubator/app-backplane)

� Definition of a set of common building blocks for web applications

A practical implementation of the W3C

Multimodal Architecture.

W3C Multimodal Architecture.

Proof of Concept Implementation.

� Distributed implementation using SCXML, HTML and CCXML/VXML

Data Component

InteractionManager

DeliveryContext

ComponentSCXML

Deutsche Telekom Laboratories Nov 19, 2008 22

Modality Component 1

Modality Component N

Runtime Framework

Modality Component API Modality Component API

HTMLfor GUI

SCXML

VoiceXMLfor VUI

W3C Multimodal Architecture.

Proof of Concept Implementation.

� Apache commons SCXML interpreter + ECMAScript interpreter.

� VoiceXML/CCMXML platform and HTML browser with AJAX support.

Server

Apache commons SCXML

Deutsche Telekom Laboratories Nov 19, 2008 23

HTML Browser(with AJAX support)

CCXML/VoiceXMLPlatform

Modality Component API: HTTP + XML (using AJAX) Modality Component API: HTTP + XML (EMMA)

Client

SIP Phone Client

Server

Telephony interface (SIP)

GUI modality component Voice modality component

HTTP I/O Processor(Apache Tomcat)

Conclusion.

W3C Multimodal Architecture.

Conclusion.

� Multimodal architecture specification (work in progress)

� Loosely coupled architecture

� Allows for co-resident and distributed implementations

� Leverages existing W3C standards (HTML, SVG, …)

Deutsche Telekom Laboratories Nov 19, 2008 25

� EMMA 1.0 specification (annotation of user input) completed

� SCXML will be a candidate for Interaction Manager implementations

Thank you for your attention.

Deutsche Telekom Laboratories Nov 19, 2008 26

Ingmar Kliche

Deutsche Telekom Laboratories

10589 Berlin

Goslarer Ufer 35

Tel.: +49 30 3497 2348

Email: ingmar.kliche@telekom.de