Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB Paderborn, Germany

Cooperative Computing &Communication Laboratory

Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB

Paderborn, Germany

Multimodal Dialog Description for Mobile Devices

© Siemens AG and Paderborn University 2004

Motivating Example

Control and monitoring of electronic devices

WinampMediaplayer

PC User Interface

Problem: UI does not scale for other platforms / devices / modalities

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


MIRS - Multimodal Interaction and Rendering System

Dialog and Interface Specification Language - DISL• Structural model

Definition of generic widgets (no specific modality)(inherited from UIML with ideas from Paterno, Plomp...)

• Control/Dialog modelVariables, Rules, Actions, Events(inherited from Dialog Specification Notation – DSN)

UI Spec(DISL)

S-DISL

XSLT

HW &User

Profile

S-DISL Interpreter

abstract to concretemapping

RendererRendererRendererRendererRenderer

Apps

Server

(Mobile)Client

httptcp

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


Dialog and Interaction Specification Language - DISL

Modification of UIML Interface Section

• structure - widget organization

• Generic widget (modality independent widget)• New generic widget elements

• style - properties

• defines properties of generic widgets

• behavior - user interaction

• defines behavior for generic widgets• extension for advanced state-oriented dialog specification• states• rules• events• communication

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


1. Container Elements- variablefield- textfield- genericfield- widgetlist

- command- confirmation- genericcommand

2. Interaction Elementsb. Misc

- variablebox- textbox- genericbox- choice

a. Input

DISL – Structure Elements

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


DISL – Structure/Style/Behavior

<structure> <widget id="Play" generic-widget="command"/> ...</structure><style> <part generic-widget id="Play" >

<property id="title">Title</property><property id="description">

This is a Remote Control for Audio Playing Devices</property><property id=“help“>Press Button to Start Playing</property><property id=“visible“ >true</property><property id=“selected“ >false</property><property id=“activated“>false</property>

</part></style><behavior> ...</behavior>

<interface ...>

</interface>

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


DISL - Behavior<interface id=“MediaPlayer" state=“init">… <behavior>

<variable ...> ... </variable> ...

<rule ...> <condition>...</condition>

<event...> ...</event></rule> ...<transition ...>

<if-true .../><action>

<statement ...> </statement>...

</action></transition>

... </behavior></interface>

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


Events:TogglePowerSetVolumeToggleMutePlayTitleStopTitleTogglePauseNextTitle...

States:Power {#isOn,#isOff}Volume {#0,#1,…,#100}Mute {#MuteOn,#MuteOff}Playing {#PlayingOn,#PlayingOff}Paused {#PauseOn,#PauseOff}...

Dialog Specification Notation (DSN)Basic Concepts

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


Rules:

Power 1. #isOn TogglePower #isOffPower 2. #isOff TogglePower #isOn

Mute 1. #isOn #MuteOn ToggleMute #MuteOffMute 2. #isOn #MuteOff ToggleMute #MuteOn

Playing 1. #isOn PlayTitle #PlayingOn #PauseOffPlaying 2. #isOn NextTitle #PlayingOn #PauseOffPlaying 3. #isOn PreviousTitle #PlayingOn #PauseOffPlaying 4. #isOn #PlayingOn StopTitle #PlayingOff #PauseOff

Paused 1. #isOn #PausedOn #PausedOff #PlayingOnPaused 2. #isOn #PausedOff #PlayingOn #PausedOn #PlayingOff

Dialog Specification Notation (DSN)Basic Concepts (cont‘d)

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


States: Power {#isOn,#isOff}Events: TogglePower

DSN to DISL

<rule id="isOn"> <condition> <equal> <variable id=“Power">true</variable> </equal> </condition></rule>

<variable id="Power" type="boolean">false</variable>

<rule id="TogglePower"> <condition> <equal> <property generic-widget="TogglePower" id="selected">true</property> </equal> </condition></rule>

<rule id="isOff"> <condition> <equal> <variable id=“Power">false</variable> </equal> </condition></rule>

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


Rules:

Power 1. #isOn TogglePower #isOff

DSN to DISL

<transition> <if-true rule-id="TogglePower" expression="and"> <if-true rule-id="isOn"/> </if-true> <action> <call source="http://localhost:8080/mediaplayer/servlet" id="tryoff" synchronized="yes" timeout="5s"> <parameter id="request"> <value-of>turnoff</value-of> </parameter> </call> <statement> <variable-content id=“isOn“><const>false</const></variable> </statement> </action></transition>

2 conditions to be true

2 actions to execute

- http request with 1 parameter

- 1 variable assignment

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


<transition> <if-true ...> ... </if-true> <action> <interface id="VolumeControl"/> </action></transition>

DISL – Additional Features

Change of Interface at Runtime

More Complex Expressions<condition> <op operator=“and“> <op-expression expression=“greaterthan“> ... </op-expression> <op-expression expression=“lessthan“> ... </op-expression> </op></condition>

<transition> <event id=“checkstatus“ repeat=“yes“ time=“5s“> <action> <call source=„...“ synchronized="yes" timeout="5s"> </action> </event></transition>

Periodic Events

Content of virtually any element can be modified

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


Current Implementation

UI Spec(DISL)

S-DISL

XSLT/Saxon ProcessorS-DISL Interpreter

J2ME/MIDP1.0

AWT-Renderer(J2ME/

MIDP1.0)

PCTomcat

http-server

http

kXML Parser

Siemens M55

Now with Bluetooth support!!

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion


Summary & Future WorkSummary

• XML-based specification language for general UIs• advanced state-oriented dialog specification

by variables, rules, actions, events(basic concepts inherited from Dialog Specification Notation – DSN)

• definition of generic widgets NOT for a specific modality, e.g., text, graphics, speech(basic concepts inherited from UIML, Paterno, Plomp...)

• Implementation for Java enabled phones

Future Work• use shorthand vocabulary• Better structuring through object orientation (step from DSN to

ODSN)• Cooperation with high level modelling techniques (bridging the gap

between e.g. task models and rendered UI) • Proof of concept with more modalities (Speech and Gesture)

Motivation

MIRS

DISL

Implemen-tation

Summary &Conclusion

Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB Paderborn, Germany

Documents

Transcript of Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB Paderborn, Germany