Measuring Human Multimodal Behaviorand

Developing Multimodal Interfaces

Jean-Claude MARTIN

LIMSI-CNRS, FRANCE

LINC, University Paris 8, FRANCE

martin@limsi.fr

www.limsi.fr/Individu/martin

2

• Is it possible to identify a “minimal set” of “generic” types of cooperation between communication modalities that• are independent of the modalities• are observed in human behavior• should be used in computer I/O

behavior

3

CONTENT

1. Framework2. Measuring Behavior3. Futures applications

4

1. FRAMEWORKTYCOON (Martin 93)

5

1. FRAMEWORKDefinitions

6

2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer et al. 98)

• How to analyze the multimodal behavior ?

• How to specify a multimodal system ?

• Find a common grid ?• 10 annotated subjects

7

2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer et al. 98)

GRAPHICS Royal ontario museum selected

SPEECH Can you give me more information about the museum

GESTURE -

8

2. MEASURING BEHAVIORSRI WOZ Toronto Map

• [Kehker et al. 98]

Form Total

UnselectedSelected

ImmediateSelected Not

ImmediateUnselected

Selected Immediate

Selected Not Immediate

Full NP 20 32 5 10 4 0 71Definite NP 2 1 1 0 0 0 4"here" 0 0 0 5 3 0 8"there" 0 7 3 0 3 1 14"this" NP 0 0 0 2 10 0 12"that" NP 0 1 0 0 0 0 1"this" 0 4 0 8 5 0 17"they" 0 1 0 0 0 0 1"it" 0 6 0 0 2 0 8empty : 'Info" 0 22 2 13 1 0 38Total 22 74 11 38 28 1 174

No Gesture Simultaneous gesture

9

2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer et al. 98)

GRAPHICS Campbell House not selected but name displayed

SPEECH What is there in the Campbell House?

GESTURE Draws an arrow towards Campbell House

• Annotating and managing ambiguity• Annotate cooperation at a finer level• Nb modalities, nb of attributes

10

2. MEASURING BEHAVIORIST-ISLE Project

• 2000-2002• Natural Interaction and MultiModality • NIS, URome, IMS, DFKI, MPI, ELRA, LIMSI• Survey on Data Resources

• 22+5 on gestures + speech• Survey on Coding Schemes

• 10+3 on gestures + speech

http://isle.nis.sdu.dk/

11

2. MEASURING BEHAVIORXML DTD + Java

12

2. MEASURING BEHAVIORXML DTD + Java

For each object oFor each reference r

If the target of r is oThen sum += salienceOf o in r

Rate red-compl = sum / nbReferences

13

3. FUTURE DIRECTIONSApplication to teaching

• How do teachers modify their mm communication behavior depending on student level, time slots …

• Towards mm pedagogical agents• BQR Université Paris 8

14

3. FUTURE DIRECTIONSSpeech Therapist Corpus

• Multiple tasks• Goals

• Who types ?• Improvement

• Multimodal• Mediated• Collective gesture

15

3. FUTURE DIRECTIONSApplication to home automation

• Goals• Usability of mm for in-home application

(interactive TV) • Applying / Adapting the TYCOON methodology

• RNRT Project• 2 years• Partners :

• MediaLive (simulation room of in home environment)• Laboratoire Conception de Produits Nouveaux et Innovation

(Ecole Nationale Supérieure d’Arts et Métiers)

Measuring Human Multimodal Behaviorand

Developing Multimodal Interfaces

Jean-Claude MARTIN

LIMSI-CNRS, FRANCE

LINC, University Paris 8, FRANCE

martin@limsi.fr

www.limsi.fr/Individu/martin