“Using eyes, hands and brain for 3D interaction with...

1

1Anatole Lécuyer, HDR defense, June 18th 2010

Habilitation à diriger des recherches

“Using eyes, hands and brain for 3D interaction with virtual environments:

a perception-based approach”

Anatole Lécuyer

Habilitation defense, June 18th 2010, INRIA/IRISA Rennes

2


Virtual reality?

Strange and controversial terminology, numerous definitions

Source of fascination, phantasm and fear

Strong and well defined scientific field

Numerous and real applications

« Virtual reality is dreams. » Morton Heilig

I-O Display Systems

NASA

3


Virtual Reality (VR)

Definition : “a virtual reality system is an immersive system that provides the user with a feeling of presence (the feeling of "being there" in the virtual world) by means of plausible interactions with a synthetic 3D environment simulated in real-time”.

Virtual reality interfaces• Visual displays : stereoscopic 3D display• Haptic interfaces : force and tactile feedback • Brain-Computer Interfaces : control with brain activity

IMMERSION Graz Tech. Univ.Univ. of Illinois

4


VR applications :• Medicine (surgical training, reeducation)• Industry (maintenance operations, data visualization)• Entertainment (video games, theme parks)• Arts and design (interactive sketching, CAD, architecture reviews)

VR challenges

1. Hardware 2. Software 3. Interaction Techniques 4. Perception

Mechanics,

Electronics, etc

Computer

Science

Ergonomics,

Computer-Human Interaction

Neuroscience

psycho/physiology

TRANSVERSALITY

5. VR Applications

Researchactivity

5


Research objective

Improve 3D interaction with virtual environmentsMaking full use of available interfaces and sensory modalities:Visual, haptic and brain-computer interfaces>> “using eyes, hands and brain”

Open questions1. Sensory feedback : more immersive2. Interaction technique : more efficient

Perception-based approach• Use knowledge in human perception• For design and evaluation • Collaborations : Univ. Paris 5 (J.M. Burkhardt), Collège de France (A.

Berthoz), Univ. Pierre Mendès-France (E. Gentaz), Freiburg Univ. (J. Wiener), INSERM (O. Bertrand, J.P. Lachaux), etc

6


Overview of research activity

1. Novel approaches for visual and haptic feedback of VE• Visual feedback based on user’s gaze• Haptic feedback : Spatialized Haptic Rendering• Combination of visual and haptic feedback : Pseudo-Haptic Feedback

2. Optimal integration of visual and haptic interfaces in VR• Software architecture : multimodal rendering of contacts• Hardware configuration : influence of spatial delocation• Visuo-haptic interaction techniques : Haptic Hybrid Control

3. Toward brain-based interaction with VE• Signal-processing techniques for BCI• Interaction techniques based on BCI• Evaluations of BCI use• Performance models for BCI

7


Vision

Virtual environment

Haptic Interface

Visual Interface

Visuo-haptic integrationBrain

Haptic feedback

Interaction TechniquePseudo-haptics

Visual attention

Motor action

Touch

Visual feedback

Mental activity Brain-Computer Interface

(1) Sensory feedback(2) Integration of visual and haptic interfaces

(3) Brain-based interaction

Research framework

8



1. Novel approaches for visual and haptic feedback of VE• Visual feedback: based on user’s gaze• Haptic feedback : Spatialized Haptic Rendering• Combination of visual and haptic feedback : Pseudo-Haptic Feedback


3. Toward brain-based interaction with VE• Signal-processing techniques for BCI• Interaction techniques based on BCI• Evaluations of BCI use• Performance models for BCI >> papers, manuscript

9



1. Novel approaches for visual and haptic feedback of VE• Visual feedback: based on user’s gaze• Haptic feedback : Spatialized Haptic Rendering• Combination of visual and haptic feedback : Pseudo-Haptic Feedback



10


Vision

Virtual environment

Haptic Interface

Visual Interface


Haptic feedback


Visual attention

Motor action

Touch

Visual feedback


Visual feedback

11


Visual feedback of virtual environments

Computer Graphics• Long history of research [Foley95] [Watt99] [Shirley05]• Impressive results :

synthetic images ~ real images

Interactive 3D graphics and VE>> Real-time, interactivity constraints• Augmented computation capacities [Pharr05]• Interactive visual effects [Guitton95] [Drettakis97]• Perception-based rendering [Mulder00] [Devlin05]

Classical loop • Visual feedback based on user’s actions • >> Head motions, hands actions

12


Novel visual feedback loop1. Gaze tracking2. Visual feedback based on perception3. Automatic adaptation of visual feedback to user’s gaze

Visual feedback based on user’s gaze

1

2

3

13


Step 1: Gaze-tracking in VE

Existing methods :

Hardware = gaze-trackers>> Numerous systems [Glenstrup95] [Bohme06]• Intrusive vs remote technology• (+) efficient,• (-) expensive, cumbersome

Software = visual attention models>> Numerous models [Lee09] [Itti98] [Longhurst06]• Bottom-up component (color, depth, motion, etc)• Top-down component (memory, habituation,

spatial context, etc)• (+) no hardware,• (-) not real-time, not designed for 3D VE

[Yee01]

[Hillaire08]

14


Step 1: Combination of gaze-tracking and visual attention models

Objective : improve gaze-tracking• Combine and associate both technique• Improve overall performance

Method Read gaze tracking output (pixel P) Compute uncertainty window (Wu) Use visual attention model inside Wu Select the most salient pixel (new P)

Evaluation• Comparison with gaze tracker alone• Improve performance of low-cost tracker

(Hillaire et al., Computer Graphics Forum, 2010)

15


Objective : simulation of visual perception properties

Effect : Depth-of-Field visual blur

Implementation : • Lens model [Potmesil81] • Auto-focus zone• GGPU technique

Evaluation : • Interactive, real-time effect• Well appreciated (Hillaire et al., ACM VRST, 2007)(Hillaire et al., IEEE CG&A, 2008)

Step 2: Perception-based visual feedback

16


Step 3: Automatic adaptation of visual effects to user’s gaze

>> Results: strong subjective preference (Hillaire et al., IEEE VR, 2008)

17






18


Vision

Virtual environment

Haptic Interface

Visual Interface


Haptic feedback


Visual attention

Motor action

Touch

Visual feedback


Combination of visual and haptic feedback

19


Pseudo-haptic feedback

Novel approach for haptic (visuo-haptic) feedback

Initial idea (Lécuyer et al., IEEE VR, 2000)• Simulate haptic feedback without a haptic interface• Compatible with passive input devices (mouse)• Use visual feedback to generate haptic illusions

Concept refinement (Lécuyer, Presence, 2009)• ”Pseudo-haptic feedback corresponds to the perception of a haptic

property that differs from the physical environment, by combining visual and haptic information and proposing a new coherent representation of the environment.“

Logitech

20


Example: pseudo-haptic textures

Objective: Feel the texture of an image with a mouse

Method• Change cursor’s motion (speed)• C/D gain function of height/depth• Patent (2004)

Experimental evaluation• Ability to perceive bumps/holes• Fine perception

Unchanged motion of the cursor

Decelerated motion

Bump (as displayed on the screen, i.e. in top-view)

Unchanged motion of the cursor

Accelerated motion

(Lécuyer et al., ACM SIGCHI, 2004)

Green cursor

White mask

21


History

Foundation = my PhD Thesis (1998-2001)

Students = PhD (Dominjon, Bibin) and Master (Tan)

Collaborations = Univ. Paris 5, UPMF, CLARTE, AFPA, etc

Dissemination = tutorials on « Perception-based haptic rendering » (EUROHAPTICS 2006, IEEE VR 2007, IEEE VR 2008)

Active field = University of British Columbia, Lund University, Fraunhofer, etc

30+ papers, recent survey (Lécuyer, Presence, 2009)

22


Overview of research and applications

Studies on haptic properties• Friction (IEEE VR 2000)• Stiffness (IEEE VR 2001)• Mass (IEEE VR 2005)• Textures (ACM CHI 2004) (Eurohaptics 2010)

Studies on applications• SAILOR medical simulator (ACM VRST 2008)• Virtual Technical Trainer « VTT » (IEEE VR 2005) (Eurohaptics 2004)

23


Lessons learned

A novel approach for visuo-haptic feedback in VE• Specificity of pseudo-haptic feedback

– Different from : « real » haptics, sensory substitution, tangible interfaces

• Perceptual phenomenon – Illusion or not?– Inter-individual variability– Necessary tuning and calibration

Numerous applications• Pseudo-haptic method?

– Use of input devices (preference for elastic devices)– Use of Control/Display gain

24






25


Vision

Virtual environment

Haptic Interface

Visual Interface


Haptic feedback


Visual attention

Motor action

Touch

Visual feedback


Integration of visual and haptic interfaces

26


Combination of visual and haptic interfaces

Spatial problem : Different sizes of haptic and visual workspaces • Haptic workspace is smaller• Example : a PHANToM in a CAVE

Hardware solutions1. Scale 1 haptics [Borro04] [Dominjon07]2. Wearable haptics [Nitzsche03]>> no simple solution

Software solutions1. Scaling techniques [Fischer03]2. Clutching techniques [Johnsen71]>> trade-off = precision / time

>> novel approach : Haptic Hybrid Control

[Fischer03]

Haptic Visual

CEA

27


Context: point-based manipulations (translations, 3DOF haptics)

Method• Manipulation workspace = « Bubble »• Hybrid Position/Rate Control• Visual and Haptic display of boundary

The « Bubble » technique

(Dominjon et al., Visual Computer, 2007)(Dominjon et al., Worldhaptics, 2005)

Rate control

Position control

28


Bubble technique

Evaluation• Setup : PHANToM in a CAVE• Task : Painting a 3D model• Conditions : clutching, scaling, bubble

Results• Faster, more precise, preferred

(Dominjon et al., LNCS, 2006)

29


Haptic Hybrid Rotations

Context : 3D object manipulations (rotations)

Method : DOF separation > cone (2DOF), springs (roll DOF)

Results : fast, precise, preferred (Dominjon et al., IEEE VR, 2006)

30


Haptic Hybrid Control

Generic approach for visuo-haptic setup• Hybrid position/rate control• Visual display of the boundary between the two control zones • Haptic display of the boundary with force-feedback

2 Implementations• Bubble (translations)• Haptic Hybrid Rotations (rotations)

Other studies• Tactile pad, 2D Desktop [Casiez07]• Pen, hand-held computers [Hachet08]

Technological transfer • VIRTUOSE API (Haption Company)

(Dominjon et al., Visual Computer, 2007)

31






32


Vision

Virtual environment

Haptic Interface

Visual Interface


Haptic feedback


Visual attention

Motor action

Touch

Visual feedback


Brain-based interaction

33


Brain-Computer Interfaces (BCI)• Novel input interface [Vidal73] [Wolpaw02]• Mental tasks : Motor Imagery (MI), attention signals, etc

Limited previous work in BCI and VR • BCI for VR : basic tasks [Bayliss03] [Lalor05]• VR for BCI : learning/motivation [Friedman07]

Problem : small number of (robust) commands

Challenges(1) Neuroscience/electrophysiology studies(2) Peripherals and mental sensors(3) Brain signal processing techniques(4) Novel and more adapted interaction paradigms

Brain-computer interfaces and VR

(Lécuyer et al, IEEE Computer, 2009)

[Friedman07]

[Leeb07]

34


BCI-based interaction techniques

Problem : small number of mental commands

Sate-of-the-art in Navigation [Scherer08]• 3 mental states : Left/Right hand, Feet MI• 3 commands : turn left/right, advance

Novel concept : high-level orders• 3 mental states : Left/Right hand, Feet• Selection of target point (binary tree)• Automatic transportation

Evaluation• Task : Navigation• 2 conditions : SoA [Scherer08], High-level

Results • High-level orders faster that soa

(Lotte et al., Presence, 2010)

[Scherer08]

35


BCI evaluationProblem : limitations of current evaluations

• Few subjects, lab conditions, intense training

Need for large-scale studies [Guger03]• Assess BCI usage and potential• Real-life conditions (limited learning, out the lab)

Large-scale evaluation (n = 21)• Entertaining application : “use the force”• Lift a virtual spaceship with feet MI

Results• Real ~50 % success • Imagined ~25 % success• Importance of feedback

Cité des sciences (since May)

(Lotte et al., BCI Workshop, 2008)

36


OpenViBE software

Integration platform • Open-source software • Design, test and use BCI• Real-time processing of cerebral data

ANR Projects• OpenViBE (2005-2009), OpenViBE2 (2009-2012)

Manpower• 12 men.year, • 3 developers (full-time)

Diffusion• 1st release in May 2009• Numerous users • http://openvibe.inria.fr

(Renard et al., Presence, 2010)

http://openvibe.inria.fr/�

37


Perspectives

38


Summary of results

Perception-based rendering of virtual environments• Interactive visual feedback based on user’s gaze: (1) novel

techniques for gaze tracking, (2) novel visual effects such as depth-of-field blur

• Novel visuo-haptic technique: “pseudo-haptic feedback” using vision to distort haptic perception

>> Perspectives• Novel visual attention models: more adapted to VE (taking into

account VR interaction), simulating eyes movements (saccades, smooth pursuits, etc). Novel visual effects : realistic camera motions.

• Investigation of pseudo-haptic phenomenon: other modalities (auditory?), perceptual perspective/understanding (integration model? use of other tools: neuroimagery, electromyography, etc)

39


Summary of results

Optimal combination of visual and haptic interfaces• Novel interaction paradigm: “Haptic Hybrid Control” to solve

problems related to spatial discrepancies

>> Perspectives• Study of temporal delay : perceptual integration models (Postdoc Nizar

Ouarti, collaboration LPPA/Alain Berthoz)? Design guidelines?• Next generation of visuo-haptic interfaces/hardware

40


Summary of results

BCI-based interaction with virtual environments• Novel 3D interaction techniques based on high-level orders to make

up for small number of commands• Large-scale BCI evaluations assessing the usage and potential of BCI

>> Perspectives : • Hybrid BCI : « combining rather than selecting », with multiple signal

processing and multiple brain signals• Advanced sensory feedback : using VR for improved motivation,

closing the loop with multi-sensory rendering

41


Technological transfer

Design of VR applications/prototypes • Medical simulator, aeronautics maintenance,

vocational training, assistance to disabled, etc

Patents • (1) pseudo-haptics, (2) haptic motion

Softwares • OpenViBE, SAMIRA (EADS/Airbus),

Multimodal rendering platform (CEA)

Industrial collaborations (PhD/projects) • Orange Labs, CEA, UBISOFT, etc

Transfer of interaction techniques • « Haptic Hybrid Control » (Haption

company), « Magic Barrier Tape » (CEA) (Sreng et al., IEEE TVCG, 2006)(Sreng et al., ACM VRST, 2007)

(Lécuyer et al., IEEE VR, 2003)

42


Long-term perspectives

Incorporate other modalities..• Body motion, Feet :

interaction and feedback• « Natural Interactive Walking »

EU Project (2009-2011)• PhD theses: Gabriel Cirio,

Léo Terziman

Towards 3D interaction based on perception and cognition..• Implicit BCI : automatic

adaptation of interaction• OpenViBE2

ANR Project (2009-2012)• PhD thesis Laurent George

43


Acknowledgements (local)

Colleagues : Bruno ARNALDI, Alain CHAUFFAUT, Rémi COZOT, Stéphane DONIKIAN, Georges DUMONT, Thierry DUVAL, Fabrice LAMARCHE, Maud MARCHAL, Julien PETTRE,

Postdoc : Marco CONGEDO, Zhan GAO, Nizar OUARTI, Tony REGGIA-CORTE, Mingjun ZHONG,

PhD Students : Gabriel CIRIO, Lionel DOMINJON, Laurent GEORGE, Sébastien HILLAIRE, Fabien LOTTE, Jean SRENG, Léo TERZIMAN,

Expert engineers : Lazar BIBIN, Laurent BONNET, Vincent DELANNOY, Yann JEHANNEUF, Yann RENARD, Aurélien VAN LANGHENHOVE,

Master students : Cédric ARROUET, Laurent ETIENNE, Thomas ERNEST, Jean-Marie HENAFF, Ludovic HOYET, Olivier JOLY, Jildaz LEBILLER, Tristan LE BOUFFANT, Jozef LEGENY, Taegi LIM, Bruno RENIER, Jean-Baptiste SAUVAN, Julien SUPPO, Chee-Hian TAN, Sebastien THOMAS, Maxime VIGNON,

44


Vision

Virtual environment

Haptic Interface

Visual Interface


Haptic feedback


Visual attention

Motor action

Touch

Visual feedback


(1) Sensory feedback(2) Integration of visual and haptic interfaces

(3) Brain-based interaction

Lionel Dominjon (2004-2007)

Sébastien Hillaire(2007-)

Fabien Lotte (2005-2008)

Jean Sreng(2005-2008)

Acknowledgements

45


Thank you

Questions?

46


Habilitation à diriger des recherches

“Using eyes, hands and brain for 3D interaction with virtual environments:

a perception-based approach”

Anatole Lécuyer

Habilitation defense, June 18th 2010, INRIA/IRISA Rennes

47


Novel visual attention models

Experimental approach : analysis of gaze behavior when navigating in VE• Navigation task : corridor with turns • Gaze recording (gaze-tracker)• Gaze analysis and modelling

Analysis results • Gaze behavior similar in VE: Anticipation of turns [Grasso98],

saccades [Berthoz00], etc

Novel visual attention model• Based on angular velocity • High performance

(Hillaire et al., ACM VRST, 2009)

48


Perception-based visual feedback

Objective : Simulation of visual flow and ocular reflexes during walk

Model : Oscillating camera motions

Innovation : Compensation for constant focalization

Results : • Interactive, real-time effect• Subjective preference for sensation of walking• Improvement of perception of traveled distance

(Lécuyer et al., IEEE VR, 2006)(Terziman et al., IEEE VR, 2009)

49


Visual feedback based on gaze

Objective: automatic adaptation of visual effects to user’s gaze

Method Extraction of user’s gaze Filtering Modification of visual effects using gaze data:

– Adaptation of camera motions– Adaptation of depth-of-field blur

Results• Subjective preference (Hillaire et al., IEEE VR, 2008)

50


Pseudo-haptic feedback

Assertions1. Pseudo-haptic feedback implies one or more sensory conflicts between visual

and haptic information.2. Pseudo-haptic feedback relies on sensory dominance of vision over touch for

spatial properties.3. Pseudo-haptic feedback corresponds to a new, and coherent, representation of

the environment resulting from a combination of haptic and visual information.4. Pseudo-haptic feedback may create haptic illusions, i.e., the perception of a

haptic property different from the one present in the real environment.

51


Pseudo-haptic textures: follow-up

Second technique : the “size technique”

Concept : change size of cursor (zoom-in and out)

Evaluation• 5 experiments• Comparison with speed effect• Combination and Conflict

Results• Size dominates speed• Combination performs better

Hole

displayed on the screen (i.e. in top-view)

Disk-shape cursor

Video Bump

Video Conflict (Lécuyer et al., ACM TAP, 2008)

52


Visuo-haptic set-up• Widely spread• Applications

Hard challenges• Software issues : common models, global architecture, interoperable or

abstract components.• Hardware issues : spatial and temporal discrepancies between

workspaces, joint usages

Research activity• Software architecture

for visuo-haptic simulations• Integration of visual and haptic

hardware in VR systems

Combination of visual and haptic interfaces

53



Context : 6DOF Manipulation of objects (rotations)

Method• Hybrid position/rate control• DOF separations • Visual display = cone (2DOF) and springs (roll DOF)• Haptic display = Force-Feedback • Emulation of elastic input device

(Dominjon et al., IEEE VR, 2006)

54



Evaluation• Setup : Virtuose and monitor• Task : Building pyramid of cubes• Conditions : scaling, clutching, HHR

Results• Faster, more precise, preffered

0

50

100

150

200

250

300

scaling clutching hybrid

total task completion time (s)

(Dominjon et al., IEEE VR, 2006)

55


Multimodal rendering of contact

Generic software architecture for multimodal rendering of contact

Examples of multisensory cues• Visual feedback of contacts : glyphs and lights• Auditory feedback of contacts : spatialized sound

(Sreng et al., IEEE TVCG, 2006) (Sreng et al., ACM VRST, 2007)

56


Haptic Rendering of Contact Position Using Vibrations

Context : 6DOF Haptic rendering of complex industrial simulations

Objectives• >> Improve perception of impacts in VE• >> Provide contact ‘position information’

Vibration patterns • 6DOF high-frequency force transient• Example : beam model

(Sreng et al., EuroHaptics, 2008)

Force

Time

57


Examples of vibration patternsAm

Fr

AmFr

AmCFr

58


6DOF extension of concept

« Virtual beam metaphor »

Experimental approach

Promising technique**

Spatialized Haptic Rendering(Sreng et al., IEEE VR, 2009)

59


Optimal integration of visual and haptic interfaces

Visuo-Haptic setups:

Strong differences• Temporal delay• Spatial mismatch

Challenges• Perceptual Studies• Novel interfaces and

techniques

Research activitySpatial discrepencies:1. Translational offset

(Congedo et al., Presence, 2006)2. Size difference

60


Delocation of visual and haptic interfaces

Delocation: spatial offset between vision and touch

Experimental approach• Perceptual study• Psychophysical experiment• Two conditions : collocated vs delocated• Computation of sensory weights

Results • Weight of vision increases

with delocation

Design guidelines• Strong impact on VR setups• Collocation to promote haptics(Congedo et al., Presence, 2006)

61


Improving EEG signal processing

Measurement of brain activity

1 Sensor : ElectroEncephaloGraphy (EEG)

0 500 1000 1500 2000 250050

0

50

C3

0 500 1000 1500 2000 250050

0

50

C4

0 500 1000 1500 2000 250050

0

50

P3

0 500 1000 1500 2000 250020

0

20

P4

0 500 1000 1500 2000 250020

0

20

O1

0 500 1000 1500 2000 250020

0

20

Sample (250 per second)

O2

Brain measures (EEG)

62




Preprocessing

Feature extraction

1

2

3

Novel approach using inverse solutions

(Lotte et al., IEEE TNSRE, 2007)(Lotte et al., PMB, 2006)

• Temporal information [Schlogl97] [Penny99]• Frequency information [Pfurtscheller01] [Palanappian05]

63




Preprocessing

Classification

Feature extraction

1

2

3

4Results :

• Better performances• Novel capacities : physical meaning, readability, rejection, etc

Linear classifiers, neural networks, nearest neighbor, probabilistic classifiers [Lotte07] [Bashashati07 ]

(Lotte et al., J. Neur. Eng., 2007)

(Lotte et al., IEEE TNSRE, 2007)

(Zhong et al., PRL, 2008)

(Lotte et al., ICPR, 2008)

Survey of existing approaches

Testing novel approaches• Fuzzy Inference Systems :

• Gaussian Process :

• Reject Options :

64


Iterative research approach:

Design and evaluation

Research activity1. Interaction techniques

– Small number of commands

2. Evaluations– Performance,

preference, usages3. Performance models

– Predict performance, improve design

BCI-based interaction with VE

65


Step 3: BCI Performance models

Objective : predict performance and improve design of techniques

“P300-signal” applications• Visual attention on flashing objects (letters, icons, virtual objects) • Object selection triggers a state change [Rebsamen07]

Performance model based on Markov Chains• Time to select a target• Number of flashes needed

Experimental validation• Three different techniques:• Good results: matching

between model and experimental data

(Sauvan et al., ACM CHI, 2009)

“Using eyes, hands and brain for 3D interaction with...

Documents

Transcript of “Using eyes, hands and brain for 3D interaction with...