Université catholique de Louvain (UCL) Belgian Laboratory of Computer-Human Interaction (BCHI)...

August 24th, 2004

WCC'2004 - Topical day on Multimodal interaction

1

Université catholique de Louvain (UCL)Belgian Laboratory of Computer-Human Interaction (BCHI)Place des Doyens, 1B-1348 Louvain-la-Neuve (Belgium)

Presented by

Multimodality and context-aware Multimodality and context-aware adaptationadaptation

Quentin Limbourg, Jean Vanderdonckt

August 24th, 2004


2

Context-awarenessContext-awareness

What is in the context of use?– User– Platform– Environment

• Organisation• Socio-psychological factors

Two examples of adaptation due to context-awareness– Adaptability = adaptation by the user– Adaptivity = adaptation by the system

August 24th, 2004


3

Example of adaptabilityExample of adaptability

Personalisation– Requires many editing– Is hard to achieve– Does not necessarily factor out common parts– Does not necessarily separate fixed parts from variable parts

Goal– To have a system for automated generation of UIs

• With support of personalisation• That can generate multiple variants of the same UI

– For this purpose, we use• A feature model• Generative Programming

August 24th, 2004


4

CC

F1F1

CC

F1F1 F2F2

CC

F1F1 F2F2

DependenciesDependencies

1.1.

2.2.

What‘s a Feature Model?What‘s a Feature Model?

OptionalOptional ExclusiveExclusive

AlternateAlternate

August 24th, 2004


5

Example of a Feature ModelExample of a Feature Model

August 24th, 2004


6

Feature modelFeature model

CC

F1F1 F2F2 F3F3

F2aF2a F2bF2b F3aF3a F3bF3b

CC

F1F1 F2F2 F3F3

F2aF2a F2bF2b F3aF3a F3bF3b

0 1 1

1 10 0

Model SpecificationsModel Specifications

<C> <F1>0</F1> <F2>1 <F2a>1</F2a> <F2b>0</F2b> </F2> <F3>1 <F3a>0</F3a> <F3b>1</F3b> </F3></C>

<C> <F1>0</F1> <F2>1 <F2a>1</F2a> <F2b>0</F2b> </F2> <F3>1 <F3a>0</F3a> <F3b>1</F3b> </F3></C>

USIXMLSpecifications

USIXMLSpecifications

August 24th, 2004


7

Generative ProgrammingGenerative Programming

void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;

ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);


ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);V

oid channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;

ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);

public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT<T>&); virtual ~ImgProcessT();

virtual char* getClassName() const; virtual char getClassID () const;

ImgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&);

ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL);mgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&);

ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL);

V

oid channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;




ImgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&);

ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL);mgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&);

ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL);





US

IXM

L-S

p eci

f ica

t ion

US

IXM

L-S

p eci

f ica

t ion

ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL) ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL);

ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL)

ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL) ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL);

ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL)

Components void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, Pl*), const int& nAngle, Pl* ppl );

void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(Pl*), Pl* ppl );

void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, const int&, const int&, const int& ), const int&, const int&, const int&, const int& );

void channelShowHistogram( int*, const int&, const int&, const int&, const int&, const int&, const int&, const int& n = 0 ) const;

void channelFft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL ); void channelIfft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL );





ImgProcessT<T>& operator+=(const ImgProcessT<T>&);I

void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, Pl*), const int& nAngle, Pl* ppl );

void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(Pl*), Pl* ppl );

void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, const int&, const int&, const int& ), const int&, const int&, const int&, const int& );

void channelShowHistogram( int*, const int&, const int&, const int&, const int&, const int&, const int&, const int& n = 0 ) const;

void channelFft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL ); void channelIfft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL );





ImgProcessT<T>& operator+=(const ImgProcessT<T>&);I

Final codeImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL);

ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);

public:

ImgProcessT(const char* const psz = "", Pl* ppl = NULL);

ImgProcessT(const int&,

const int& nWidth = 20,

const int& nHeight = 20,

const int& nFillColor = 0,

ProgressLine* ppl = NULL

);

ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL);

ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);

public:

ImgProcessT(const char* const psz = "", Pl* ppl = NULL);

ImgProcessT(const int&,

const int& nWidth = 20,

const int& nHeight = 20,

const int& nFillColor = 0,

ProgressLine* ppl = NULL

);

void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL);

void channelRange (T*, Pl* ppl = NULL );















August 24th, 2004


8

Example of adaptivityExample of adaptivity

Computer-aided surgery by mixed reality

August 24th, 2004


9

Multi-platformMulti-platform

Extending approaches for multiple contexts of use– Context is now restricted

to one computing platformat a time

Final User Interface

Physical User Interface

Logical User Interface

Task & Concepts

Platform 1:Web terminal

August 24th, 2004


10


Traditional approach: last level only



Platform 2:Personal computer


Platform 3:Pocket PC


August 24th, 2004


11


Traditional approach: Atomica

HTML/Java UI code


Platform 2:Personal computer

Visual C++ UI code


Mobile Visual Basic UI

August 24th, 2004


12


Shortcomings of the traditional approach (ct’d)– Repetition of development and maintenance

[UIML,1999]


Platform 2:PC


Java UI #1 C++ UI #1 Mobile UI #1Application #1



August 24th, 2004


13


Physical level deployment

Platform 1:



Platform 2:



Platform 3:



= Translation relationship= Transformation at the same level of abstraction

August 24th, 2004


14


Physical level deployment: Galaxy

Windows UI code


Platform 1:Personal Computer

Platform 2:Macintosh Mac OS

Macintosh UI code


Platform 3:Workstation OSF/Motif

OSF/Motif UI code


August 24th, 2004


15


Physical level deployment: Galaxy

Windows UI code



Macintosh UI code


OSF/Motif UI code

August 24th, 2004


16


Logical level deployment













August 24th, 2004


17


Logical level deployment: UIML

HTML UI code

HTML Renderer

UIML Specification

Platform 1:Personal Computerwith Web browser

Platform 2:Personal ComputerWith XML browser

XML UI code

XML Renderer

Platform 3:Mobile Phone

WML UI code

WML Rendered

August 24th, 2004


18

<?xml version="1.0" ... ?><uiml version="2.0">

</uiml>

Multi-platform:Multi-platform:UIML Skeleton – Part 1UIML Skeleton – Part 1

What parts comprise the UI &what’s their relationship?

<interface> <structure> <part class=“Button”> … </part>

... </structure>

</interface>

[Abrams et al., 1999]

August 24th, 2004


19


Task & Concepts level deployment




Task & Concepts

Platform 1 Platform 2




Task & Concepts

Platform 3




Task & Concepts

August 24th, 2004


20





Task & Concepts





Task & Concepts

Platform 3




Task & Concepts

Multiple levels of translation/reification[Thevenin,2001]

August 24th, 2004


21


Translation at Task & Concept level




Task & Concepts





Task & Concepts

August 24th, 2004


22


Translation at Task & Concept level– Task model primitives

• Sub-tasks (nodes)– Remove sub-task: for different reasons– Add sub-task: unlikely to appear– Modify sub-task: switch, reshuffle, relax constraints

• Hierarchical & temporal operators (edges)– Modify temporal operator: reordering– Cutting sub-trees

– Concepts model primitives

August 24th, 2004


23

Register to CHI’2001 Conference

Pay registration

By credit card By fax

[ ]

RecordPayment mode

>>

Specify all registration needs

Enter personal information Specify special needs Specify attendee type

| | | | | |

>>


Task translation

| | |

August 24th, 2004


24


Concepts removal: CT-UIMS[Märtin,1990]

August 24th, 2004


25





Task & Concepts





Task & Concepts

Translations T&C + logical levels

August 24th, 2004


27





| | | >>

August 24th, 2004


28


Translations T&C +logical levels

August 24th, 2004


29





Task & Concepts





Task & Concepts

Translations T&C + logical + physical levels

August 24th, 2004


30


Translations T&C + logical + physical levels– T&C + logical remain constant over time or

coordinated with changes– Graceful degradation of presentation

• Widgets: replacement, reduction, resizing• Text: text summarization, sentence elision,

automatic summary• Images: color reduction, cropping, resizing,…• Animation: reduction to still images,…• Video: reduction of frame, refreshing,…

August 24th, 2004


31


Graceful degradation of presentation– Widgets

August 24th, 2004


32

Multi-Multi-platformplatform Graceful

degradation of presen-tation– Images

August 24th, 2004


33


Graceful degradation of presentation– Images

August 24th, 2004


34





Task & Concepts





Task & Concepts

Translations combined at all levels

August 24th, 2004


35


Multiple transformations possible




Task & Concepts





Task & Concepts

August 24th, 2004


36


Plastic UI = UI that can accomodate multiple changes of context (here, platforms) while preserving some usability properties




Task & Concepts

Platform 1 Platform 2 Platform 3

Final User Interface Final User Interface

[Calvary, Coutaz & Thevenin, 2001]

August 24th, 2004


37


Plastic UI: Virtual keyboard[Grolaux, 2001]

August 24th, 2004


38


Plastic UI: FlexClock[Grolaux, Van Roy, Vanderdonckt, 2002]

August 24th, 2004


39


Plastic UI: adaptable bounded value

August 24th, 2004


40


Reverse engineering




Task & Concepts





= Abstraction relationship= Transformation into amore abstract level ofabstraction

ReversiXML

WebRevenge

ReversiXML

August 24th, 2004


41

XIML2WML generatorXIML2WML generator

August 24th, 2004


42


Reverse engineering





= Crossing relationship= Transformation betweendifferent levels ofabstraction

Logical User InterfaceRetargeting

[Bouillon & Vanderdonckt,2002]

August 24th, 2004


43

Multi-directional User InterfacesMulti-directional User Interfaces

A UI is represented by a suite of models that are– Analysable– Editable– Transformable

Transformational approach– Any UI is subject to any transformation at any time

• From one language to another• From one computing platform to another• From reverse engineering to forward engineering• …

August 24th, 2004


44

Multi-directional User InterfacesMulti-directional User Interfaces

Model engineering– Forward engineering (top-down)– Reverse engineering (bottom-up)– Bidirectional engineering (both ways)

Model transformations– Reification, abstraction– Translation– Crossing (retargeting, revamping,…)

August 24th, 2004


45

ContextsContextsof useof useChannelsChannelsModalitiesModalities

Multiple levels of considerationMultiple levels of consideration

Channels plus:Channels plus:

• User behaviourUser behaviour• User expectationsUser expectations• User experienceUser experience• User groupsUser groups• Physical constraintsPhysical constraints• EnvironmentEnvironment

Modalities plus :Modalities plus :

• Market segmentsMarket segments• Time of usageTime of usage• Place of usagePlace of usage• Network, Wi-fiNetwork, Wi-fi• Various tasksVarious tasks

PlatformsPlatformsDevicesDevices

Platforms plus :Platforms plus :

• Modalities of Modalities of interactioninteraction

• Mono vs Mono vs multimodalitymultimodality

Devices plus :Devices plus :

• Software Software characteristicscharacteristics

• Hardware Hardware characteristicscharacteristics

• Languages: Languages: programming, markupprogramming, markup

• Tools: renderer, Tools: renderer, interpreter, compilerinterpreter, compiler

August 24th, 2004


46

ContextsContextsof useof useChannelsChannelsModalitiesModalities

USIXMLUSIXML

PlatformsPlatformsDevicesDevices

August 24th, 2004


47

An overview of USIXML toolsAn overview of USIXML tools

Running prototypes– GrafiXML: CUI Hi-Fi editor + Code Generator (Java

Swing, XHTML)– FormiXML: CUI Hi-Fi editor for Forms (XForms, HTML)

– SketchiXML: CUI Lo-Fi Sketching– VisiXML: CUI Mid-Fi (MS Visio Plug-in)– FlashiXML: CUI renderer for SVF– ReversiXML: reverse engineering from HTML to CUI,

AUI– TransformiXML API: transformation tool– Teresa (F. Paternò, CUI level to multi FUI generation)

Demo

Demo

Demo

August 24th, 2004


48

Thank you very much for your attention!Thank you very much for your attention!

www.usixml.org

Credits

www.plasticity.org

www.similar.cc

Université catholique de Louvain (UCL) Belgian Laboratory of Computer-Human Interaction (BCHI)...

Documents

Transcript of Université catholique de Louvain (UCL) Belgian Laboratory of Computer-Human Interaction (BCHI)...