Enaction and the Concept of Presence

Enactive/06

Structure• Introduction (10 minutes)

– Presentation of the Symposium (6 minutes)– Presentation of the panelists (1 minute for each panelist)

• Discussion sessions around 4 questions– Answer of each panelist (2 minutes for each panelist)– Questions from public and answers from panelists (8

minutes)

• Conclusions– Summing up of the relevant points emerged (10

minutes)

Introduction

Aim• The aim of this special session at Enactive/06 is to explore

relationships between Enaction and Presence. • The invited panellists are members drawn from the

Enactive Network of Excellence and the Peach FET Coordination Action in Presence.

• They will debate fundamental concepts, philosophical roots and practical examples of enactive interfaces and presence.

Basic concepts: Enaction & Presence

• Enactive knowledge is constructed on motor skills and doing is the mean for learning in an enactive context (Bruner, 1966). Enactive interaction is direct, natural and intuitive and enactive interfaces deal with the role played by action in the shaping of the perceptual content, the role of active exploration and the role of perception in the guidance of action. The close coupling of the perception-action loop is hence a key characteristic of enactive interfaces.

• Research into the concept of presence also draws upon similar notions. Presence has been defined as ‘the illusion of non-mediation’ (Lombard and Ditton, 1997). It is concerned with developing technologies that make it possible for people to feel as if they are somewhere else, or to feel that distant others are in fact close by. It is concerned with tele-presence and haptic interaction, with new places for interaction, with gesture, full body and other forms of multimodal interactions.

• Many issues concerning Enaction and Presence are still under debate and will addressed by 4 main questions.

Questions1. Believability is a notion that has been explored in the domains

of literature, theatre, film, and animated characters or agents. Believability has been conceived of as the illusion of life or of reality, the illusion that even the odd, caricatured, unrealistic characters are alive or real.

But the mental state of illusion of reality is far for being conceptually and psychologically well characterized. It can be understood as an illusion of non-mediation, as transparency. But it is not evident that users and film or other media audience mistakenly take a fake world for real nor that believability require this error. People confronted with VR worlds know they are experiencing something that is not real, yet they may react as if it were real.

Can we consider Believability as one condition that makes users and spectators act and react as if the virtual world was real? In this case, which actions and reactions should be taken into account as manifestations of believability?

Questions2. The notion of realism as simulation of the real world is problematic

too.

The Japanese roboticist Mori has introduced the notion of the Uncanny Valley: the sense of familiarity increases in association to realistic robotic artefacts (human-like robots) up to a certain point and then comes to a valley of negative familiarity and strangeness. Out of the valley there is another, higher peak of positive familiarity associated with perfectly human-like artefacts and real humans. The fall in the valley of uncanny is especially experienced in presence of discrepancies between visual cues and cues from other sensory modalities and between the realism of the physical aspect and a lower level of realism for behavioural aspects.

Is photorealism enough to faithfully simulate the physical characteristics of the real world or should we be concerned with other sensory modalities? Is realism as simulation of the properties of the real world a necessary condition for producing believable worlds and agents? Which is the role of coherence?

Questions

3. Feelings of connectedness come through presence allowing interfaces that provide a high degree of presence to extend a person’s influence.A surgeon performing a tele-operation wants to feel as if he or she is working on a body immediately to hand. The surgeon wants to make movements and gestures as they would in the real world.

What meanings can gestures have at the technological scale and at the human scale?

Questions

4. The sense of immersion is also important to a sense of presence.

But in mixed realities the human is effectively immersed in both the real and the digital condition.

How can realism, consistency and believability be maintained as people move between representations?How can we design to make interactions with digital artefacts as natural and intuitive as interactions with the physical world with which we are so familiar?

Panelists

David Benyon: Background

Framework 5 Presence project, Benogo (Being there, no need to go)– we were concerned with reproducing real places through

using photo-realistic images. – Issues of fidelity of representation were dealt with– wanted to capture the sense of place experienced– design for presence as a sense of place.

• Good interaction is unmediated interaction– Presence as the illusion of non-mediation– Enaction as the tight coupling of perception and action– Interacting through rather than interacting with

• Embodied Interaction– Paul Dourish ‘Where the action is’– Malcolm McCullogh ‘Digital Ground’– Marshall McLuhan Media as the ‘extensions of man’

• Mixing Realities• Designing for interaction at a human scale

Position Statement

Wijnand IJsselsteijn, Eindhoven University of Technology

Enactive /06Montpellier, 20 nov 2006

Imagine the year 12,006....

What would a team of Martian paleantologists and archeologists deduce about our species when encountering the remnants of our information society, i.e., a desktop computer?

Homo Desktopus

© Koert van Mensvoort

‘El Cine del Futuro’ – Morton Heilig (1955)

Embodiment?

“The biggest challenge to developing telepresence is achieving that sense of “being there.” Can telepresence be a true substitute for the real thing? Will we be able to couple our artificial devices naturally and comfortably to work together with the sensory mechanisms of human organisms?” – Marvin Minsky, 1980

Teleoperation

Fluent telemanipulation

When we interact with virtual or remote environments using intuitive interaction devices, isomorphic to our sensorimotor abilities, the real-time, reliable and persistent chain of user action and system feedback allows the integration of technology as a phenomenal extension of the self and the perception of technology-mediated environments as distal, but actionable environments in relation to one’s own active probing.

On the road to transparancy….

George Papagiannakis

• Position Statement. Mixed Realities (MR) and their concept of cyber-real space interplay invoke such interactive ’touching’ experiences that promote new patterns of believability and presence. Believability is a term used to measure the realism of interaction in the MR environments. Case studies are represented by the reproduction of ancient frescos-paintings through the real-time revival of their fauna and flora, featuring groups of virtual animated characters with artificial life dramaturgical behaviours in an immersive, fully mobile Augmented Reality environment. The main goal is to push the limits of current Augmented Reality (AR) and virtual storytelling technologies and exploring the processes of mixed narrative design of fictional spaces (e.g. frescos-paintings) where visitors can experience a high degree of realistic immersion. We argue that in order to deliver ‘real’ experiences and to create believable interactive applications for mobile MR, future steps in mobile MR enabling technologies should cater for enhanced social awareness of the virtual humans to the real world and new channels for interactivity between the real users and virtual actors. Only then the believability factor of virtuality structures will be enhanced and allow for compelling real experiences through mobile virtual environments.

Thomas A. Stoffregen

• Position statement. There seems to be a powerful desire to equate the quality of simulation of virtual environment systems with their ability or tendency to induce certain subjective experiences in users, such as the feeling of “being there”. Many concepts used in the study and design of virtual environment systems are not rigorously defined, and have no widely accepted definition (e.g., presence, immersion, fidelity). Weak definitions make it difficult to use such concepts for the formulation or testing of empirical hypotheses about user performance. Weak concepts also make it difficult to determine what factors truly are important in evaluating the quality of simulation and virtual environment systems. In some applications (e.g., entertainment), the subjective experience of users is the chief criterion for system quality. However, in many other applications, the subjective experience of users is irrelevant, or of only indirect relevance to the quality of the system. In many applications, it is the user’s behavioral interaction with the system that matters most, and the success of this behavioral interaction often can be addressed through direct measures of behavior, rather than though indirect measures of subjective experience.

Answers

Question 1 Believability

• Answer sequence: – George – David– Thomas– Wijnand

George Papagiannakis: Main elements of believability

Elements of BelievabilitySensory Level

Believability

PerceptualLevel

BelievabilityVirtual World

Participants

Interface

Immersion

Presentation

Interaction

EmotionEmotion

PersonalityPersonality

Physical Info.Physical Info.

EmotionEmotion

PersonalityPersonality

Physical Info.Physical Info.

Elements of BelievabilitySensory Level

Believability

PerceptualLevel

Believability

Sensory Level

Believability

PerceptualLevel

BelievabilityVirtual World

Participants

Interface

Immersion

Presentation

Interaction

Immersion

Presentation

Interaction

EmotionEmotion

PersonalityPersonality

Physical Info.Physical Info.

EmotionEmotion

PersonalityPersonality

Physical Info.Physical Info.

EmotionEmotion

PersonalityPersonality

Physical Info.Physical Info.

EmotionEmotion

PersonalityPersonality

Physical Info.Physical Info.

George Papagiannakis• Personality and

Emotion for VHs• Facial animation

with emotion • Body animation

with emotion: – Idle motions– Gestures– Reflex motions

• Integration and blending

• Face & body are both important for perceiving emotions and for realism of motion

David Benyon: Believability

• Scenarios are very powerful in terms of believability

• E.g. In security guard scenario people ‘heard’ glass breaking when there was none.

• In an auralisation of a large hall people ‘heard’ footsteps

• Many famous illusions rely on a false scenario

David Benyon: Believability

• Believability is about the combination of media, narrative, affect, activities

• Presence enables us to fill in the gaps of experience; to create believable experiences.

• It is about what is not said, or not shown that makes us believe

• In literature, the gap in the text

Thomas Stoffregen: Belief in what?

• Two types of subjective experience: Subjective Realism versus Subjective Reality

• Subjective Realism– Degree to which a virtual environment resembles the

corresponding real environment– User correctly believes that they are interacting with a

simulation• “That looks realistic”, or “that looks believable” implies “I know it is not

real”

• Subjective Reality– The virtual environment cannot be distinguished from the

corresponding real environment– User incorrectly believes that they are in the real

environment• These are mutually exclusive

Thomas Stoffregen: Design goals

• When do users need Subjective Realism?– When is it useful for users to know that they

are interacting with a simulation?

• When do users need Subjective Reality?– When do we want users to believe that they

are truly in the depicted environment?

• In most cases, we want subjective realism, not subjective reality

• We need to be clear about which experience users need

Question 2 Realism

• Answer sequence: – Thomas– Wijnand– George– David

Thomas Stoffregen: Realism and Believability

• Can we have a strong definition of realism or believability?

• My claim: Realism/believability is context-specific, and is influenced by experience– When linear perspective was invented (ca. 1400), it

was called “realistic”– When photography was invented (1839), it was called

“realistic” (and linear perspective was called “unrealistic”)

– When cinema was invented (ca. 1890), it was called “realistic” (and photography was called “unrealistic”)

– When closed-loop displays were invented (ca. 1970), they were called “realistic” (and cinema was called “unrealistic”)

Thomas Stoffregen: Obsolescence

• A technology that is “realistic” or “believable” today may be “primitive” or “unrealistic” tomorrow

• There is no fixed meaning, no fixed standard or criterion for realism, or for believability

• Conclusion: Realism and believability should not be general criteria for design

vs

George Papagiannakis:Believable virtual characters

• Virtual Characters composited via offline Special Effects (SFX) in movie industry

George Papagiannakis Illumination registration for believable virtual characters

A. Egges, G. Papagiannakis, N. Magnenat-Thalmann. An Interactive Mixed Reality Framework for Virtual Humans. Cyberworlds 2006, EPFL, Switzerland, November 28-29, IEEE Computer Society. September 2006

David Benyon: Realism• In a comparison of photo-realistic and graphical

virtual worlds, there was little difference

David Benyon: Realism

• Comparing real and photo-realistic renderings of places, responses to real places were more detailed, more exaggerated

David Benyon: Comparing Real and Virtual

• Comparing real and virtual

Very Quite Neither Quite Very

Attractive 23 3 1 1 1 Ugly

Big 7 12 6 2 1 Small

Colorful 5 12 8 4 Colorless

Noisy 4 8 5 8 4 Quiet

Temporary 1 7 6 8 7 Permanent

Available 4 11 10 4 Unavailable

Versatile 2 11 8 7 1 Limited

Interactive 5 8 5 6 5 Passive

Pleasant 23 5 2 Unpleasant

Interesting 19 6 2 1 1 Boring

Stressful 1 1 3 4 20 Relaxing

Very Quite Neither Quite Very

Attractive 7 15 7 Ugly

Big 2 11 13 2 1 Small

Colorful 14 5 9 1 Colorless

Noisy 3 7 9 6 4 Quiet

Temporary 3 6 8 8 4 Permanent

Available 1 9 11 5 2 Unavailable

Versatile 2 6 12 7 2 Limited

Interactive 1 7 6 10 5 Passive

Pleasant 4 16 5 4 Unpleasant

Interesting 8 11 5 4 Boring

Stressful 5 5 11 8 Relaxing

David Benyon: Realism

• Stereo effects were very important to realism

• The uncanny valley exists in places as well as characters– A slightly wrong

perspective– A strange field of view

• Non-realism can be exploited for interesting experiences

• Presence of the non-realistic - the sense of ‘being there’ in au unreal place - can be powerful.

Question 3 Connectedness

• Answer sequence: – David– Thomas– Wijnand– George

David Benyon: Connectedness

• Presence is not a cognitive phenomena; it is about embodied interaction

• Wearing a head mounted display (HMD) and you lose your hands and feet!

• Current VR is inherently disembodied - need for more sensors and full body interaction.

David Benyon: Connectedness

• As remote control and extensions of physicality.– The blind person’s stick

• Need to increase influence of haptics in interactions

• Connectedness is also about connecting people with people: social presence– Phones, connected rings/earrings, glowing

picture frame

Thomas Stoffregen: Tele-Surgery

• Do we want the surgeon to feel as if they are working on a real body?

• Or, do we want the surgeon to complete the surgery successfully?

• These are not necessarily the same• If we can get “successful performance”, then “it

feels real” may be irrelevant

Thomas Stoffregen: Believability and action

• Two concepts of believability, or fidelity• 1. Experiential Fidelity

– What subjective experiences are associated with an interface or virtual environment

• Does the system “feel real”?

• 2. Action Fidelity– Do users’ actions in a virtual environment

resemble their actions in the corresponding real environment?

• Was the operation a success?

Thomas Stoffregen: Empirical question

• Experiential Fidelity may not imply action fidelity• Action Fidelity may not imply experiential fidelity• Research is needed to understand how they are

related– Does experiential fidelity provide a significant

improvement in performance?

Thomas Stoffregen: Specific applications

• If Action Fidelity and Experiential Fidelity are different, then we need to ask– In what applications should we try to design for

Experiential Fidelity (when do we want a virtual environment to “feel real”?)

• Example: Entertainment applications (e.g., video games)– In what applications should we try to design for

Action Fidelity (when do we want users to move in a virtual environment as the move in the real world?

• Example: Process control (e.g., control rooms)– In what applications do we want both types of

fidelity?• Example: Vehicle simulators?• It is possible to have both Experiential and Action fidelity

Figure 1: Real and virtual interactive characters in MR

Figure 3: Close up of AR HMD & camera h/w components

Figure 2: Overview of believable animation process for improved presence and interaction in MR

George Papagiannakis: Connectedness

Question 4 Immersion

• Answer sequence: – Wijnand– George– David– Thomas

Panorma Mesdag – View of Scheveningen, 1881

Presence experiment at GeorgiaTech

M. Botvinick (2004) Science, Vol 305, Issue 5685, 782-783

(2) This hand is my hand…

George Papagiannakis: Believability and Immersion in

MR• Mobile Mixed Reality Presence Environments

– Dramaturgical Storytelling Virtual Human Simulation in AR (LIFEPLUS EU Project: http://www.miralab.unige.ch/subpages/lifeplus)

• Personalized learning and training environments– Training-Interactivity in MR populated Industrial

Environments (STAR EU Project: http://review.realviz.com/)

- [Papagiannakis05b] G.Papagiannakis, S. Schertenleib, B. O’Kennedy , M. Poizat, N.Magnenat-Thalmann, A. Stoddart, D.Thalmann, "Mixing Virtual and Real scenes in the site of ancient Pompeii", Computer Animation and Virtual Worlds, p 11-24, Volume 16, Issue 1, February 2005

- [Papagiannakis05a] G. Papagiannakis, H. Kim, N. Magnenat-Thalmann, "Believability and Presence in Mobile Mixed Reality Environments", IEEE VR2005 Workshop on Virtuality Structures, February 2005

David Benyon: Immersion

• Breaks in presence and Fragments of presence

• Immersion for how long?

• More important than believability and realism.

• The Book ‘Problem’ in Presence research

David Benyon: Immersion• Attention and divided attention• The mixed reality of car driving.• Maintaining immersion (attention?) when moving

from one experience to another.

Thomas Stoffregen: Mixed-Reality

• All interfaces and virtual environments involve mixed reality

• This is especially clear in the context of multisensory perception

• Stimulation of the vestibular, haptic, and kinesthetic systems always is influenced by the “real world”

Conclusions