Post on 01-Feb-2016
description
July 13th, 2009 Marco Blumendorf I Marco.Blumendorf@dai-labor.de
Multimodal Interaction in Smart Environments
A Model-based Runtime System for Ubiquitous User Interfaces
Dipl.-Inf. Marco Blumendorf
Fakultät IV – Elektrotechnik und InformatikTechnische Universität Berlin
A Model-based Runtime System for Ubiquitous User Interfaces 2July 13th, 2009
Agenda
• Motivation– Current Trends
• Fundamentals– User Interface Models
• Approach– Runtime Concepts– UI Description Language Extensions– Architecture
• Implementation– Does it work?
• Conclusion– Some issues remain
A Model-based Runtime System for Ubiquitous User Interfaces 3July 13th, 2009
“Recent” Trends
World Wide Web & Ubiquitous Computing
A Model-based Runtime System for Ubiquitous User Interfaces 4July 13th, 2009
World Wide Web
A Model-based Runtime System for Ubiquitous User Interfaces 5July 13th, 2009
World Wide Web - Basics
• Websites hosted on servers
• Accessible from everywhere
• HTML language
• Browser as HTML interpreters
A Model-based Runtime System for Ubiquitous User Interfaces 6July 13th, 2009
Web Page Development in 4 Steps
1. Write HTML code
2. Add some CSS (style)
3. Add some JavaScript
4. Deploy it on a webserver
Model,interpreted by the browser
A Model-based Runtime System for Ubiquitous User Interfaces 7July 13th, 2009
Ubiquitous Computing
A Model-based Runtime System for Ubiquitous User Interfaces 8July 13th, 2009
Computers are everywhere!
A Model-based Runtime System for Ubiquitous User Interfaces 9July 13th, 2009
“Ubiquitous Computing”(Mark Weiser, 1991)
“In the 21st century the technology revolution will move into the everyday,
the small and the invisible.”
A Model-based Runtime System for Ubiquitous User Interfaces 10July 13th, 2009
Smart Objects Smart Devices
Smart NetworksSmart Environments
A Model-based Runtime System for Ubiquitous User Interfaces 11July 13th, 2009
Interaction happens everywhere!
A Model-based Runtime System for Ubiquitous User Interfaces 12July 13th, 2009
Ubiquitous User Interfaces(UUIs)
A Model-based Runtime System for Ubiquitous User Interfaces 13July 13th, 2009
Utilizing Ubiquitous User Interfaces in Smart Environments
used interaction modalities
A Model-based Runtime System for Ubiquitous User Interfaces 14July 13th, 2009
Compared to a web page …
How would you develop a Ubiquitous User Interface?
A Model-based Runtime System for Ubiquitous User Interfaces 15July 13th, 2009
Language and Infrastructure
for
Ubiquitous User Interfaces
A Model-based Runtime System for Ubiquitous User Interfaces 16July 13th, 2009
Agenda
• Motivation We need a language and infrastructure for UUIs
• Fundamentals– User Interface Models
• Approach– Runtime Concepts– UI Description Language Extensions– Architecture
• Implementation– Does it work?
• Conclusion– Some issues remain.
A Model-based Runtime System for Ubiquitous User Interfaces 17July 13th, 2009
User Interface Models
A Model-based Runtime System for Ubiquitous User Interfaces 18July 13th, 2009
HTML as a Model
User Interface
HTML HTML
A Model-based Runtime System for Ubiquitous User Interfaces 19July 13th, 2009
Models to Maximize Common Code
User Interface
Model
A Model-based Runtime System for Ubiquitous User Interfaces 20July 13th, 2009
UIDLs use Multiple Levels of Abstraction
Abstract Interface
Concrete Interface
Final User Interface
Concepts and Task Model
See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003)
Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004)and TERESA (Berti et al. 2004)
A Model-based Runtime System for Ubiquitous User Interfaces 21July 13th, 2009
UIDLs use Multiple Levels of Abstraction
Abstract Interface
Concrete Interface
Final User Interface
Concepts and Task Model
See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003)
Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004)and TERESA (Berti et al. 2004)
A Model-based Runtime System for Ubiquitous User Interfaces 22July 13th, 2009
What if you want to change anything at runtime?
A Model-based Runtime System for Ubiquitous User Interfaces 23July 13th, 2009
You need the models!
A Model-based Runtime System for Ubiquitous User Interfaces 24July 13th, 2009
Problems
• Current approaches have a design time focus• At runtime, knowledge is hidden within code• Design intentions are not explicit anymore• Design decisions can not be revised
• This makes UIs static and limits adaptivity!
=> No Ubiquitous User Interfaces
A Model-based Runtime System for Ubiquitous User Interfaces 25July 13th, 2009
Keep the UI models at runtime!
A Model-based Runtime System for Ubiquitous User Interfaces 26July 13th, 2009
Model Interpretation
A Model-based Runtime System for Ubiquitous User Interfaces 27July 13th, 2009
Model Interpretation
A Model-based Runtime System for Ubiquitous User Interfaces 28July 13th, 2009
Model Interpretation
A Model-based Runtime System for Ubiquitous User Interfaces 29July 13th, 2009
Model Interpretation
A Model-based Runtime System for Ubiquitous User Interfaces 30July 13th, 2009
Agenda
• Motivation We need a language and infrastructure for UUIs
• Fundamentals We need models interpretation at runtime
• Approach– Runtime Concepts– UI Description Language Extensions– Architecture
• Implementation– Does it work?
• Conclusion– Some issues remain.
A Model-based Runtime System for Ubiquitous User Interfaces 31July 13th, 2009
• Executable Models
• UI Description Language
• Server Architecture
Framework Building Blocks
A Model-based Runtime System for Ubiquitous User Interfaces 32July 13th, 2009
How to make Models Executable?
• Preserve design information at runtime
• Add state information
• Add execution logic
A Model-based Runtime System for Ubiquitous User Interfaces 33July 13th, 2009
DefinitionElement
SituationElement
ExecutionElement
Extending Current Design Models
Dynamic Executable Model
See e.g. Breton & Bézivin 2001 for details
A Model-based Runtime System for Ubiquitous User Interfaces 34July 13th, 2009
A Task Tree Example
Abstract TaskInteraction TaskTemp. Operators>> []
A Model-based Runtime System for Ubiquitous User Interfaces 35July 13th, 2009
Metamodel of Executable Task Models
A Model-based Runtime System for Ubiquitous User Interfaces 36July 13th, 2009
Metamodel of Executable Task Models
DefinitionElement
SituationElement
ExecutionElement
A Model-based Runtime System for Ubiquitous User Interfaces 37July 13th, 2009
Metamodel of Executable Task Models
DefinitionElement
SituationElement
ExecutionElement
A Model-based Runtime System for Ubiquitous User Interfaces 38July 13th, 2009
Metamodel of Executable Task Models
DefinitionElement
SituationElement
ExecutionElement
A Model-based Runtime System for Ubiquitous User Interfaces 39July 13th, 2009
• Executable Models
• UI Description Language
• Server Architecture
Framework Building Blocks
A Model-based Runtime System for Ubiquitous User Interfaces 40July 13th, 2009
How to build a language for such models?
• Extend existing models with runtime concepts– Add state & execution logic– Synchronization of models through mappings– Interaction with external entities
• Additional concepts for Ubiquitous User Interfaces– Multimodal and distributed interaction– Context-sensitivity– Adaptation
A Model-based Runtime System for Ubiquitous User Interfaces 41July 13th, 2009
Mediating between human and computer
10011010101111100100
“turn the light on”
A Model-based Runtime System for Ubiquitous User Interfaces 42July 13th, 2009
Mediating between human and computer
10011010101111100100 ?
“turn the light on”
A Model-based Runtime System for Ubiquitous User Interfaces 43July 13th, 2009
Mediating between human and computer
10011010101111100100 ?
“turn the light on”
A Model-based Runtime System for Ubiquitous User Interfaces 44July 13th, 2009
Mediating between human and computer
10011010101111100100
“turn the light on”
A Model-based Runtime System for Ubiquitous User Interfaces 45July 13th, 2009
Net of UI Models
A Model-based Runtime System for Ubiquitous User Interfaces 46July 13th, 2009
Mediation Process
A Model-based Runtime System for Ubiquitous User Interfaces 47July 13th, 2009
Adding State and Execution LogicDefinitionElement
SituationElement
ExecutionElement
A Model-based Runtime System for Ubiquitous User Interfaces 48July 13th, 2009
Synchronizing Models at Runtime
Task Model Service Model
Task Service Call
executeexecute
resultresult
Mapping
Link
Source
TargetEnabledEnabled
ApplicationApplication
A Model-based Runtime System for Ubiquitous User Interfaces 49July 13th, 2009
A UUI is defined by a set of linked & stateful executable models.
DefinitionElement
SituationElement
ExecutionElement
DefinitionElement
SituationElement
ExecutionElement
DefinitionElement
SituationElement
ExecutionElement
DefinitionElement
SituationElement
ExecutionElement
DefinitionElement
SituationElement
ExecutionElement
A Model-based Runtime System for Ubiquitous User Interfaces 50July 13th, 2009
• Executable Models
• UI Description Language
• Server Architecture
Framework Building Blocks
A Model-based Runtime System for Ubiquitous User Interfaces 51July 13th, 2009
How to connect Executable Models to the external world?
• Connect interaction resources
• Integrate context
• Execute the model
• Provide functional components
A Model-based Runtime System for Ubiquitous User Interfaces 52July 13th, 2009
Connecting Interaction Resources
A Model-based Runtime System for Ubiquitous User Interfaces 53July 13th, 2009
Integrating Context
Sensors
A Model-based Runtime System for Ubiquitous User Interfaces 54July 13th, 2009
Application Hosting
Sensors
A Model-based Runtime System for Ubiquitous User Interfaces 55July 13th, 2009
Model State
Sensors
A Model-based Runtime System for Ubiquitous User Interfaces 56July 13th, 2009
Monitoring State
Sensors
Monitoring
A Model-based Runtime System for Ubiquitous User Interfaces 57July 13th, 2009
Stimulating Model Execution
Sensors
Stimulation
A Model-based Runtime System for Ubiquitous User Interfaces 58July 13th, 2009
Processing Components
Sensors
A Model-based Runtime System for Ubiquitous User Interfaces 59July 13th, 2009
Agenda
• Motivation We need a language and infrastructure for UUIs!
• Fundamentals We need models at runtime!
• Approach Executable Models Runtime UI Models Server Architecture
• Implementation– Does it work?
• Conclusion– Some issues remain.
A Model-based Runtime System for Ubiquitous User Interfaces 60July 13th, 2009
Does it work?
A Model-based Runtime System for Ubiquitous User Interfaces 61July 13th, 2009
Implementation
The Multi-Access Service Platform (MASP) implements the presented concepts
Features• Multimodal interaction• Dynamic distribution• Adaptation
Different applications have been implemented.
A Model-based Runtime System for Ubiquitous User Interfaces 62July 13th, 2009
A Meta UI for Smart Environments
A Model-based Runtime System for Ubiquitous User Interfaces 63July 13th, 2009
The Multimodal Cooking Assistant
A Model-based Runtime System for Ubiquitous User Interfaces 64July 13th, 2009
Results
• Performance was better than originally expected.
• We can build multimodal applications that dynamically change the used modalities.
• Applications can be distributed and dynamically configured at runtime.
• A follow me mode shows automatic reconfiguration based on context information.
• Dynamic layout changes according to the distance of the user to the screen have been realized.
A Model-based Runtime System for Ubiquitous User Interfaces 65July 13th, 2009
Agenda
• Motivation We need a language and infrastructure for UUIs!
• Fundamentals We need models at runtime!
• Approach– Runtime Concepts– UI Description Language– Architecture
• Implementation– We got promising results!
• Conclusion– Some issues remain.
A Model-based Runtime System for Ubiquitous User Interfaces 66July 13th, 2009
In Summary …
… set of models provides explicit UI state
… allows flexible generation of UIs
… dynamic interpretation of user input
… server infrastructure hosts the models
… integration with the external world
… handles context and available devices
… self-contained UI models that embed state and execution logic
… utilization at runtime and easy exchange of models
Dynamic Executable ModelDefinitionElement
SituationElement
ExecutionElement
A Model-based Runtime System for Ubiquitous User Interfaces 67July 13th, 2009
Open issues and future work
• Presented extensions of existing UIDLs are not a complete UIDL for UUIs
• multi-user and multi-application scenarios have not been adequately addressed yet
• a set of tools also covering runtime aspects is needed
• bridging design- and runtime by Executable Models brings up new issues like the preservation of consistency at runtime
• integration of natural language processing and models-based development seems to be required for future success of model-based UUIs
A Model-based Runtime System for Ubiquitous User Interfaces 68July 13th, 2009
References• [Berti 2004] Silvia Berti, Francesco Correani, Giulio Mori, Fabio
Paternò, and Carmen Santoro. Teresa: A transformation-based environment for designing and developing multi-device interfaces. In ACM CHI 2004, volume II, pages 793–794, 2004. ACM Press.
• [Breton&Bézivin 2001] Erwan Breton and Jean Bézivin. Towards an understanding of model executability. In FOIS '01: Proceedings of the international conference on Formal Ontology in Information Systems, pages 70–80, New York, NY, USA, 2001. ACM.
• [Calvary 2003] Gaëlle Calvary, Joëlle Coutaz, David Thevenin, Quentin Limbourg, Laurent Bouillon, and Jean Vanderdonckt. A unifying reference framework for multi-target user interfaces. Interacting with Computers, 15(3):289–308, 2003.
• [Limbourg 2004] Quentin Limbourg, Jean Vanderdonckt, Benjamin Michotte, Laurent Bouillon, and Víctor López-Jaquero. Usixml: A language supporting multi-path development of user interfaces. In EHCI/DS-VIS, volume 3425 of Lecture Notes in Computer Science, p. 200–220. Springer, 2004.
• [Szekely 1996] Pedro A. Szekely. Retrospective and challenges for model-based interface development. In DSV-IS, pages 1–27. Springer, 1996.
• [Weiser 1991] Mark Weiser, The computer for the 21st century, Scientific American, Morgan Kaufmann Publishers Inc., 1991, 265, 66-75
A Model-based Runtime System for Ubiquitous User Interfaces 69July 13th, 2009
The End …
Thank you for your attention!
Your questions please …
Dynamic Executable Model
DefinitionElement
SituationElement
ExecutionElement