Agenda

July 13th, 2009 Marco Blumendorf I [email protected]

Multimodal Interaction in Smart Environments

A Model-based Runtime System for Ubiquitous User Interfaces

Dipl.-Inf. Marco Blumendorf

Fakultät IV – Elektrotechnik und InformatikTechnische Universität Berlin

A Model-based Runtime System for Ubiquitous User Interfaces 2July 13th, 2009

Agenda

• Motivation– Current Trends

• Fundamentals– User Interface Models

• Approach– Runtime Concepts– UI Description Language Extensions– Architecture

• Implementation– Does it work?

• Conclusion– Some issues remain


“Recent” Trends

World Wide Web & Ubiquitous Computing


World Wide Web


World Wide Web - Basics

• Websites hosted on servers

• Accessible from everywhere

• HTML language

• Browser as HTML interpreters


Web Page Development in 4 Steps

1. Write HTML code

2. Add some CSS (style)

3. Add some JavaScript

4. Deploy it on a webserver

Model,interpreted by the browser


Ubiquitous Computing


Computers are everywhere!


“Ubiquitous Computing”(Mark Weiser, 1991)

“In the 21st century the technology revolution will move into the everyday,

the small and the invisible.”


Smart Objects Smart Devices

Smart NetworksSmart Environments


Interaction happens everywhere!


Ubiquitous User Interfaces(UUIs)


Utilizing Ubiquitous User Interfaces in Smart Environments

used interaction modalities


Compared to a web page …

How would you develop a Ubiquitous User Interface?


Language and Infrastructure

for

Ubiquitous User Interfaces


Agenda

• Motivation We need a language and infrastructure for UUIs

• Fundamentals– User Interface Models



• Conclusion– Some issues remain.


User Interface Models


HTML as a Model

User Interface

HTML HTML


Models to Maximize Common Code

User Interface

Model


UIDLs use Multiple Levels of Abstraction

Abstract Interface

Concrete Interface

Final User Interface

Concepts and Task Model

See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003)

Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004)and TERESA (Berti et al. 2004)


What if you want to change anything at runtime?


You need the models!


Problems

• Current approaches have a design time focus• At runtime, knowledge is hidden within code• Design intentions are not explicit anymore• Design decisions can not be revised

• This makes UIs static and limits adaptivity!

=> No Ubiquitous User Interfaces


Keep the UI models at runtime!


Model Interpretation


Agenda

• Motivation We need a language and infrastructure for UUIs

• Fundamentals We need models interpretation at runtime





• Executable Models

• UI Description Language

• Server Architecture

Framework Building Blocks


How to make Models Executable?

• Preserve design information at runtime

• Add state information

• Add execution logic


DefinitionElement

SituationElement

ExecutionElement

Extending Current Design Models

Dynamic Executable Model

See e.g. Breton & Bézivin 2001 for details


A Task Tree Example

Abstract TaskInteraction TaskTemp. Operators>> []


Metamodel of Executable Task Models



DefinitionElement

SituationElement

ExecutionElement


How to build a language for such models?

• Extend existing models with runtime concepts– Add state & execution logic– Synchronization of models through mappings– Interaction with external entities

• Additional concepts for Ubiquitous User Interfaces– Multimodal and distributed interaction– Context-sensitivity– Adaptation


Mediating between human and computer

10011010101111100100

“turn the light on”



10011010101111100100 ?




10011010101111100100



Net of UI Models


Mediation Process


Adding State and Execution LogicDefinitionElement

SituationElement

ExecutionElement


Synchronizing Models at Runtime

Task Model Service Model

Task Service Call

executeexecute

resultresult

Mapping

Link

Source

TargetEnabledEnabled

ApplicationApplication


A UUI is defined by a set of linked & stateful executable models.

DefinitionElement

SituationElement

ExecutionElement

DefinitionElement

SituationElement

ExecutionElement

DefinitionElement

SituationElement

ExecutionElement

DefinitionElement

SituationElement

ExecutionElement

DefinitionElement

SituationElement

ExecutionElement


How to connect Executable Models to the external world?

• Connect interaction resources

• Integrate context

• Execute the model

• Provide functional components


Connecting Interaction Resources


Integrating Context

Sensors


Application Hosting

Sensors


Model State

Sensors


Monitoring State

Sensors

Monitoring


Stimulating Model Execution

Sensors

Stimulation


Processing Components

Sensors


Agenda

• Motivation We need a language and infrastructure for UUIs!

• Fundamentals We need models at runtime!

• Approach Executable Models Runtime UI Models Server Architecture




Does it work?


Implementation

The Multi-Access Service Platform (MASP) implements the presented concepts

Features• Multimodal interaction• Dynamic distribution• Adaptation

Different applications have been implemented.


A Meta UI for Smart Environments


The Multimodal Cooking Assistant


Results

• Performance was better than originally expected.

• We can build multimodal applications that dynamically change the used modalities.

• Applications can be distributed and dynamically configured at runtime.

• A follow me mode shows automatic reconfiguration based on context information.

• Dynamic layout changes according to the distance of the user to the screen have been realized.


Agenda

• Motivation We need a language and infrastructure for UUIs!

• Fundamentals We need models at runtime!

• Approach– Runtime Concepts– UI Description Language– Architecture

• Implementation– We got promising results!



In Summary …

… set of models provides explicit UI state

… allows flexible generation of UIs

… dynamic interpretation of user input

… server infrastructure hosts the models

… integration with the external world

… handles context and available devices

… self-contained UI models that embed state and execution logic

… utilization at runtime and easy exchange of models

Dynamic Executable ModelDefinitionElement

SituationElement

ExecutionElement


Open issues and future work

• Presented extensions of existing UIDLs are not a complete UIDL for UUIs

• multi-user and multi-application scenarios have not been adequately addressed yet

• a set of tools also covering runtime aspects is needed

• bridging design- and runtime by Executable Models brings up new issues like the preservation of consistency at runtime

• integration of natural language processing and models-based development seems to be required for future success of model-based UUIs


References• [Berti 2004] Silvia Berti, Francesco Correani, Giulio Mori, Fabio

Paternò, and Carmen Santoro. Teresa: A transformation-based environment for designing and developing multi-device interfaces. In ACM CHI 2004, volume II, pages 793–794, 2004. ACM Press.

• [Breton&Bézivin 2001] Erwan Breton and Jean Bézivin. Towards an understanding of model executability. In FOIS '01: Proceedings of the international conference on Formal Ontology in Information Systems, pages 70–80, New York, NY, USA, 2001. ACM.

• [Calvary 2003] Gaëlle Calvary, Joëlle Coutaz, David Thevenin, Quentin Limbourg, Laurent Bouillon, and Jean Vanderdonckt. A unifying reference framework for multi-target user interfaces. Interacting with Computers, 15(3):289–308, 2003.

• [Limbourg 2004] Quentin Limbourg, Jean Vanderdonckt, Benjamin Michotte, Laurent Bouillon, and Víctor López-Jaquero. Usixml: A language supporting multi-path development of user interfaces. In EHCI/DS-VIS, volume 3425 of Lecture Notes in Computer Science, p. 200–220. Springer, 2004.

• [Szekely 1996] Pedro A. Szekely. Retrospective and challenges for model-based interface development. In DSV-IS, pages 1–27. Springer, 1996.

• [Weiser 1991] Mark Weiser, The computer for the 21st century, Scientific American, Morgan Kaufmann Publishers Inc., 1991, 265, 66-75


The End …

Thank you for your attention!

Your questions please …

Dynamic Executable Model

DefinitionElement

SituationElement

ExecutionElement

Agenda

Documents

Transcript of Agenda