Behaviour and interaction modeling of 3D virtual objects

A Generic Framework for Executable

Gestural Interaction Models

Romuald Deshayes, Tom Mens, Philippe Palanque

Software Engineering Lab IRIT

2012-2013

Deshayes Romuald – UMONS 1 / 15

Behaviour and interaction modeling of 3D virtual objects

Table of Contents

1 IntroductionContextPaper Goal

2 The FrameworkModular and Generic FrameworkLayers and Gestures

3 Conclusion

4 Future Work

Deshayes Romuald – UMONS 2 / 15

Behaviour and interaction modeling of 3D virtual objects

Introduction

Context

Context

Different objects with different ways of interacting with them

Enhance computer with better insight in user interaction withreal and virtual objects

[Target Domain] Virtual and Augmented Reality applications suchas games, home automation and simulation

Deshayes Romuald – UMONS 3 / 15

Behaviour and interaction modeling of 3D virtual objects

Introduction

Context

Context

ContextDevelopment of Human-Computer Interaction (HCI) applicationsinvolving gestural interaction

High technical complexity (low level aspects)Low learning curve

Deshayes Romuald – UMONS 4 / 15

Behaviour and interaction modeling of 3D virtual objects

Introduction

Paper Goal

Paper Goal

GoalProvide a generic and efficient way of describing and executing softwareinvolving gestural interaction with real or virtual objects

Based on executable models

Domain Specific Modeling Language (DSML)

A DSML simplifies the creation of gesture based application

Generic framework for gestural interaction

Deshayes Romuald – UMONS 5 / 15

Behaviour and interaction modeling of 3D virtual objects

Introduction

Paper Goal

Why Modeling

Why is executable modeling appropriate for developing HCIapplications ?

Visual aspects allow to hide low level details by raising thelevel of abstraction (simplicity)Formal analysis can help reduce the amount of bugs incomplex systems (security)Behavior can be changed at runtime by editing the runningmodels (productivity)

Challengecounter critique from [SKF+94] that model-based approaches"model the previous generation of UI"

Deshayes Romuald – UMONS 6 / 15

Behaviour and interaction modeling of 3D virtual objects

Introduction

Paper Goal

Why Modeling

Why is executable modeling appropriate for developing HCIapplications ?

Visual aspects allow to hide low level details by raising thelevel of abstraction (simplicity)Formal analysis can help reduce the amount of bugs incomplex systems (security)Behavior can be changed at runtime by editing the runningmodels (productivity)

Challengecounter critique from [SKF+94] that model-based approaches"model the previous generation of UI"

Deshayes Romuald – UMONS 6 / 15

Behaviour and interaction modeling of 3D virtual objects

The Framework

Modular and Generic Framework

The framework

GoalProvide a generic and efficient way of describing and executingsoftware involving gestural interaction with real or virtual objects

How ?By creating a modular and generic framework that allows thespecification of executable models for manipulating objects withgestures

I/O independenceClient/server approach for modularityLayered architecture

Deshayes Romuald – UMONS 7 / 15

Behaviour and interaction modeling of 3D virtual objects

The Framework

Modular and Generic Framework

A Layered Architecture

A layered architectureEach layer is modeled as an ICO (formalism based onHigh-Level Petri nets)Layers communicate through events

Deshayes Romuald – UMONS 8 / 15

Behaviour and interaction modeling of 3D virtual objects

The Framework

Layers and Gestures

The Layers and the Gestures

The layersL1 detects new users, receives raw data and converts them topositions w.r.t head positionL2 create low-level gestures and detect hand state changes

MoveOpenClose

L3 combine low-level gestures with users’ state to createhigh-level gestures

SimpleDragColinearDragNonColinearDrag

Deshayes Romuald – UMONS 9 / 15

Behaviour and interaction modeling of 3D virtual objects

The Framework

Layers and Gestures

Modeling the state of the users

Layer 3 : modeling the state of the users

Deshayes Romuald – UMONS 10 / 15

Behaviour and interaction modeling of 3D virtual objects

The Framework

Layers and Gestures

Modeling the interaction with virtual objects

Layer 4 contains a differentmodel for each objectUsed to combine all existinggestures into a modeldescribing how to interactwith an object

Example : a book

Deshayes Romuald – UMONS 11 / 15

Behaviour and interaction modeling of 3D virtual objects

Conclusion

Conclusion

Advantages of resorting to executable modeling for HCI applicationdevelopment

Reduced technical complexity thanks to higher level ofabstractionDynamic execution particularly suitable for rapid prototypingof interactive applicationsCommunicating models allow to concurrently interact withmultiple objects and enable separation of concernsOur framework allows to easily specify gestural interactionwith real or virtual objects

Deshayes Romuald – UMONS 12 / 15

Behaviour and interaction modeling of 3D virtual objects

Future Work

Future Work

Current research and future workExplore heterogeneity in the layers to use the most appropriateformalism (MultiPM vs MonoPM)Develop a DSML for Layer 4Perform a controlled user study with CS students to evaluatethe productivity gain of using the framework and its usability.Code generation

Deshayes Romuald – UMONS 13 / 15

Behaviour and interaction modeling of 3D virtual objects

Future Work

Thank you

Questions ?

Deshayes Romuald – UMONS 14 / 15

Behaviour and interaction modeling of 3D virtual objects

Future Work

Bibliography I

P. N. Sukaviriya, S. Kovacevic, J. D. Foley, B. A. Myers, D. R.Olsen Jr., and M. Schneider-Hufschmidt.Model- based user interfaces: What are they and why shouldwe care?In Proc. UIST94, pages 133–135. ACM, November 1994.

Deshayes Romuald – UMONS 15 / 15