Haptic Interaction in Mixed Environments and Virtual Reality

66
Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Haptic Interaction in Virtual and Mixed Reality Environments Konstantinos Moustakas, Assistant Professor Konstantinos Moustakas, Assistant Professor Electrical and Computer Engineering Department, University of Electrical and Computer Engineering Department, University of Patras Patras

description

Event: "IEEE Day 2012" IEEE Student Branch-University of Patras 09.10.2012 "Haptic Interaction in Mixed Environments and Virtual Reality" by K.Moustakas Museum of Sciences & Technology of the University of Patras More info: http://ieee-upatras.gr/en/events/2012/123-ieee-day-2012

Transcript of Haptic Interaction in Mixed Environments and Virtual Reality

Page 1: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality Environments

Konstantinos Moustakas, Assistant ProfessorKonstantinos Moustakas, Assistant Professor

Electrical and Computer Engineering Department, University of Electrical and Computer Engineering Department, University of PatrasPatras

Page 2: Haptic Interaction in Mixed Environments and Virtual Reality

•• WhatWhat isis haptichaptic interaction?interaction?•• HapticHaptic psychophysicspsychophysics outlineoutline•• HapticHaptic interactioninteraction essentialessential processingprocessing::

Outline Haptic interactionHaptic interaction

OutlineOutline

•• HapticHaptic interactioninteraction essentialessential processingprocessing::–– CollisionCollision detectiondetection–– HapticHaptic renderingrendering

•• ApplicationsApplications•• FutureFuture researchresearch directionsdirections

Page 3: Haptic Interaction in Mixed Environments and Virtual Reality

• Wikipedia, Haptics: Haptics refers to the sense oftouch (from Greek άπτω = "I fasten onto, I touch).

• Merriam-Webster, Haptic: relating to or based onthe sense of touch

Introduction Haptic interactionHaptic interaction

HapticsHaptics

the sense of touch• Haptics is the science of applying touch

(tactile) sensation and control to interactionwith computer applications.

Page 4: Haptic Interaction in Mixed Environments and Virtual Reality

Introduction Haptic interactionHaptic interaction

HapticsHaptics

• Tactile information– The responses of

the receptors of the skin in contact with an objectan object

• Kinesthetic information– Net forces applied

to limbs

Page 5: Haptic Interaction in Mixed Environments and Virtual Reality

Introduction Haptic interactionHaptic interaction

Haptic interaction Haptic interaction -- ImportanceImportance

• The sense of touch is underestimated!• Loss of the sense of touch can cause impairments that

cannot be compensated by sight– Hand dexterity– Haptic capabilities, grasping, etc.– Walking– Walking– Limb position perception– Eating– Speaking

• In virtual environments the loss of the sense of touch can impair performance and interaction efficiency

Page 6: Haptic Interaction in Mixed Environments and Virtual Reality

HapticHaptic interactioninteraction -- ImportanceImportance

Introduction Haptic interactionHaptic interaction

Strike a match (with vision)

Normal case

7.5 sec

Anesthesia

29 sec

Page 7: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Psychophysics Haptic interactionHaptic interaction

Different types of tactile sensorsDifferent types of tactile sensors

Page 8: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Psychophysics Haptic interactionHaptic interaction

Surface encoding Surface encoding ––deformation mattersdeformation matters

Page 9: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Psychophysics Haptic interactionHaptic interaction

Spatial resolutionSpatial resolution

Page 10: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Psychophysics Haptic interactionHaptic interaction

Temporal resolutionTemporal resolution

~1 kHz constraint

while in graphics~25 Hz constraint

Page 11: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Psychophysics Haptic interactionHaptic interaction

• Tactile– Surface texture– Orientation– Shape

What do we sense?What do we sense?

• Haptic– Mass/Weight– Force– Stiffness– Shape

– Size– Pressure– Curvature– Motion

– Stiffness– Viscosity

• Thermal– Cooling– Warming

Page 12: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Rendering Haptic interactionHaptic interaction

HapticsHaptics & Environment& Environment

Page 13: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

• Haptic devices:– Tactile devices– Force feedback devices

• Exoskeletons• One point of interaction devices

Haptic devicesHaptic devices

Page 14: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Page 15: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Page 16: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Page 17: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Page 18: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Page 19: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Page 20: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Page 21: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Page 22: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Page 23: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devices VS Human SensorsHaptic devices VS Human Sensors

Haptic devicesHuman sensors

Many sensors One sensor (x N)

Direct tactile sensation Only through motion

Deformation N/A

• Haptic devices act as a filter to the human tactile/haptic sensors.

• How would you see a visual scene with such a filter?

Page 24: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Devices Haptic interactionHaptic interaction

Haptic devices VS Human SensorsHaptic devices VS Human Sensors

Page 25: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Rendering Haptic interactionHaptic interaction

Haptic vs Graphics rendering processesHaptic vs Graphics rendering processes

ViewViewTransformTransform

LightingLighting ProjectionProjection TexturingTexturingScene Scene

TraversalTraversal

GeometryGeometry RasterizerRasterizerApplicationApplication DisplayDisplay

ForceForce TactileTactileTraversalTraversal DisplayDisplay

ForceForceCalculationCalculation

Force Force SmoothingSmoothing

ForceForceMappingMapping

HapticHapticTexturingTexturing

Collision Collision DetectionDetection

Page 26: Haptic Interaction in Mixed Environments and Virtual Reality

Collision detection Haptic interactionHaptic interaction

• Collision is one of natures powerful tools:– Object interaction

• Walking• Machines are based on collisions

Collision detectionCollision detection

• Machines are based on collisions

– Sound source

• In the real world physical laws and matter take-over

• In the virtual world???

Page 27: Haptic Interaction in Mixed Environments and Virtual Reality

Collision detection Haptic interactionHaptic interaction

Collision detectionCollision detection

In no case a trivial issue!!!

Page 28: Haptic Interaction in Mixed Environments and Virtual Reality

Collision detection Haptic interactionHaptic interaction

Bounding volume hierarchiesBounding volume hierarchies

Page 29: Haptic Interaction in Mixed Environments and Virtual Reality

Collision detection Haptic interactionHaptic interaction

Bounding volume hierarchies (video)Bounding volume hierarchies (video)

Page 30: Haptic Interaction in Mixed Environments and Virtual Reality

Collision detection Haptic interactionHaptic interaction

Force feedbackForce feedback

A. Vogiannou, K. Moustakas, D. Tzovaras and M.G. Strintzis, “Enhancing Bounding Volumes using Support Plane Mappings for Collision Detection”, Eurographics Computer Graphics Forum, vol. 29, no. 5, pp. 1595-1604, August 2010.

Page 31: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Rendering Haptic interactionHaptic interaction

Haptic rendering basicsHaptic rendering basics

Page 32: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Rendering Haptic interactionHaptic interaction

Haptic rendering: Stiffness vs StabilityHaptic rendering: Stiffness vs Stability

Force Calculation Force Calculation ––Virtual wallVirtual wall

Insufficient stiffnessInsufficient stiffnessV V <<<<<<<< 00

Virtual wall Virtual wall Moving into the wallMoving into the wall

F F

timetime

K K wallwall •• ∆∆ x + B vx + B vF =F =

SolutionsSolutionsHigh frequencyHigh frequency

Virtual couplingVirtual couplingGod object God object –– virtual proxyvirtual proxy

Page 33: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Rendering Haptic interactionHaptic interaction

Force discontinuitiesForce discontinuities

Page 34: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Rendering Haptic interactionHaptic interaction

Force shadingForce shading

Page 35: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Rendering Haptic interactionHaptic interaction

• Height maps• Friction/Damping fields• Stochastic textures

Haptic texturingHaptic texturing

• Virtual textures/bumps

Page 36: Haptic Interaction in Mixed Environments and Virtual Reality

The force field haptic rendering method

Haptic rendering Haptic interactionHaptic interaction

method

K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia , vol. 14, no. 1, pp. 62-72, January 2007.

Page 37: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

From the distance field to the force fieldFrom the distance field to the force field

-di -di

di

Distance field

Force field

K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia , vol. 14, no. 1, pp. 62-72, January 2007.

Page 38: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Haptic renderingHaptic rendering• The force feedback is calculated for a given

point through:

• whereD(x) andF(x) are the values and the

∈ 3x R

• whereD(x) andF(x) are the values and the vector of the distance and force field respectively

-di -di

di

Distance field Force field

Page 39: Haptic Interaction in Mixed Environments and Virtual Reality

SQ-Map: Efficient layered collision detection and haptic renderingdetection and haptic rendering

K. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and Haptic Rendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93, January 2007.

Page 40: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

• Motivation: New method fast as the distance fields, without high memory requirements

• SQ-Map steps– Preprocessing

SQSQ--Map overviewMap overview

– Preprocessing• Object approximate segmentation• Superquadric modeling• Distance map generation

– Run-time processing• Layered collision detection (3 levels of processing)

K. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and Haptic Rendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93, January 2007.

Page 41: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

SQSQ--Map distance mapsMap distance maps

Page 42: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Collision detection: Layer 1Collision detection: Layer 1

( )

2

1

12 2

22 2

1 2 3

, ,x y z

F x y za a a

εε

εε ε

= + +

> 1

Page 43: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Collision detection: Layer 1Collision detection: Layer 1

( )

2

1

12 2

22 2

1 2 3

, ,x y z

F x y za a a

εε

εε ε

= + +

< 1

Page 44: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Collision detection: Layer 2Collision detection: Layer 2

Page 45: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Collision detection: Layer 2Collision detection: Layer 2

dDSQ

P

If d>DSQ collision is detected

Page 46: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

• Similar processing to layer 2 with two add-ons– Possibility to perform distance map

subdivision so as to increase accuracy

Collision detection: Layer 3Collision detection: Layer 3

subdivision so as to increase accuracy– The Layer 3 distance map is used that

projects also mesh vertices onto the superquadric surfaces

Page 47: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

• Lemma: – The distance map and the superquadric can

provide an exact representation of the 3D model iff function fC that maps all points of the

Accuracy analysisAccuracy analysis

model iff function fC that maps all points of the object’s surface onto the superquadric is injective.

– Proof in the text

Page 48: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Accuracy analysisAccuracy analysis

Concave region of typeQQ2

Concave region of type Q1

Page 49: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Haptic renderingHaptic rendering

Force magnitude estimation

Page 50: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Haptic renderingHaptic rendering

Force direction estimation

Can be analytically calculated

Page 51: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Haptic renderingHaptic rendering

Advantages:

• Rapid estimation of the reaction force

• Possibility to analytically process and handle the • Possibility to analytically process and handle the force feedback

Friction:

Haptic texture:

Gaussian noise

Page 52: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

ComparisonsComparisons

Page 53: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Qualitative resultsQualitative results

OBB SQ-Map, Layer 1

SQ-Map, Layer 2 SQ-Map, Layer 3

Page 54: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic rendering Haptic interactionHaptic interaction

Qualitative resultsQualitative results

Page 55: Haptic Interaction in Mixed Environments and Virtual Reality

Applications Haptic interactionHaptic interaction

• Surgical simulation

• Telemanipulation, robot-assisted surgery

Application examplesApplication examples

• Computer aided design

• Entertainment• Haptic Visualization• Applications for the

disabled• …

Page 56: Haptic Interaction in Mixed Environments and Virtual Reality

Applications Haptic interactionHaptic interaction

Haptic mapsHaptic maps

Haptic map with embedded semantic iformation

K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia , vol. 14, no. 1, pp. 62-72, January 2007.

Page 57: Haptic Interaction in Mixed Environments and Virtual Reality

Applications Haptic interactionHaptic interaction

Haptic browserHaptic browser

Page 58: Haptic Interaction in Mixed Environments and Virtual Reality

Applications Haptic interactionHaptic interaction

Virtual reality cane simulationVirtual reality cane simulation

D. Tzovaras et.al., “Design and implementation of haptic virtual environments for the training of the visually impaired, IEEE Neural Systems and Rehab. Eng., 2004

Page 59: Haptic Interaction in Mixed Environments and Virtual Reality

Applications Haptic interactionHaptic interaction

Mixed reality cane simulationMixed reality cane simulation

Virtual environment

Virtual

Magnetic sensor 1

CyberGrasp

D. Tzovaras, K. Moustakas, G. Nikolakis and M.G. Strintzis, "Interactive Mixed Reality White Cane Simulation for the Training of the Blind and the Visually Impaired", Springer Journal on Personal and Ubiquitous Computing, vol.13, no.1, pp.51-58, January 2009.

Virtual cane

Cane replica

CyberGrasp

Magnetic sensor 2

Page 60: Haptic Interaction in Mixed Environments and Virtual Reality

VR game for the hearing and the visually impaired

Applications Haptic interactionHaptic interaction

visually impaired

K. Moustakas, L. Dybkjaer, O. Aran, D. Tzovaras and N.O. Bernsen, "Communication Between Blind and Hearing Impaired People Through a Multimodal Interactive Game", IEEE Multimedia , accepted for publication.

Page 61: Haptic Interaction in Mixed Environments and Virtual Reality

Applications Haptic interactionHaptic interaction

The ideal caseThe ideal case

Page 62: Haptic Interaction in Mixed Environments and Virtual Reality

Applications Haptic interactionHaptic interaction

ArchitectureArchitecture

Page 63: Haptic Interaction in Mixed Environments and Virtual Reality

Applications Haptic interactionHaptic interaction

Game stepsGame steps

Page 64: Haptic Interaction in Mixed Environments and Virtual Reality

Applications Haptic interactionHaptic interaction

Game videoGame video

Page 65: Haptic Interaction in Mixed Environments and Virtual Reality

Future work Haptic interactionHaptic interaction

• Haptic visualization– Use the haptic channel to complement vision

in information visualization applications

• Haptic interaction with videos

Future directionsFuture directions

• Haptic interaction with videos– Highly dynamic

• Content• Interaction

– Partial input of the environment, ill-posed

• Towards a theory of haptic rendering…

Page 66: Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality Environments

Thank you for your

Konstantinos Moustakas, Assistant Professor Konstantinos Moustakas, Assistant Professor [email protected]@ece.upatras.gr

Electrical and Computer Engineering Department, University of Electrical and Computer Engineering Department, University of PatrasPatras

Thank you for your attention