Lecture 4: Perception and Cognition in Immersive Virtual

Environments

Dr. Xiangyu WANG

• As interactive technology becomes immersive, multimodal, and ubiquitous, the emphasis in immersive virtual environments shifts to facilitating perceptual and cognitive interaction

Human Information Processing Model

Human Information Processing Model (adapted from Proctor 1994)

Human Information Processing Model

• The perceptual stage includes processes that operate from the stimulation of sensory organs through the identification of that stimulation. – involved in the detection, discrimination and identification of the

displayed information. – triggers a neural signal – the ability of the brain to extract information from signal depends

on the quality of the sensory input. • The cognitive stage operates with the goal of

determining the appropriate action or response. – include the retrieval of information from memory, comparisons

among displayed items, comparison between these items and the information in memory, arithmetic operations, decision making etc.

• The action stage selects and executes an action response, which must then be translated into a set of neuromuscular commands (e.g., specific limbs)

In-class exercise

• Go to Google, • Search “interactive system”, “Immersive virtual

space” or “immersive virtual environment”, or others…

• Study it quickly for several minutes based on some video or screenshots.

• Briefly develop and explain a “Human Information Processing Model” for the selected system

• Volunteer to present to the class

Perception

• Perception is the process of acquiring, interpreting, selecting, and organizing sensory information. (from Wikipedia)

• Depth perception is the visual ability to perceive the world in three dimensions. Depth perception allows the beholder to accurately gauge the distance to an object.

Perception

• Depth can be artificially simulated by presenting on a display’s 2D surface properties of the external world that result in the sensation of depth, called depth cues.

• Depth cues can be classified as monocular or binocular. Retinal disparity is an important binocular depth cue and is caused by the fact that each of our eyes sees the world from a different point of view. These two different perspectives of the same image are then combined by the brain into a single image by means of a process called fusion. (from Wikipedia)

Head-Mounted Displays

Cognition

• The term cognition is used in several loosely related ways to refer to a faculty for the human-like processing of information, applying knowledge and changing preferences. (from Wikipedia)

Immersive VR system

Perception and Cognition in Immersive VE

• There is a need to study how the computer-generated object is represented on 3D immersive display (VE) during the design process to maximize the perceptual and cognitive benefits.

• There is a need to evaluate the impact of Virtual Reality (VR) aided design applications on user's perceptive and cognitive system.

Perception and Cognition in Immersive VE

• Virtual objects incorporating some visual cues is associated with a significant computational expense.

Perception and Cognition in Immersive VE

• In a U.S. National Institute of Health report of the National Advisory Mental Health Council, the impact of virtual reality environments on cognition was specifically cited with the recommendation that: “Research is needed to understand both the positive and negative effects of such participation on children’s and adults’ perceptual and cognitive skills”.

• This recommendation should also be used in immersive VE for design.

Case Study

• Experiment I: Depth Estimation of a 3D Object (from GAGGIOLI and BREINING 2001)

Experimental Objective

• The main purpose of the first experiment was to determine how stereo vision and different monocular coding techniques affect the ability to estimate the depth of a 3d computer-generated object.

• Depth perception can be studied by manipulating any of depth cues (e.g. visual texture, shading, stereo disparity) and determining how perceptual responses are altered.

Experimental Method

• Subjects: Twenty-four subjects served as participants in the study (mean age = 29.4 years); normal vision and no experience with virtual reality and CAD software.

• Design– The experiment consisted of two types of object

geometry (simple versus complex), – four types of graphic images (wireframe, flat shading,

Gouraud shading and Gouraud shading with surface normals)

– six depth values (15, 30, 45, 60, 75, 90 cm)

Experimental Method

• Apparatus and Stimuli– Created by software 3D Studio Max, release 2.5– Displayed on the front wall of a four-walls CAVE

capable of both stereoscopic and monoscopic modes.– Stereoscopic condition was created using

CrystalEyes time multiplexed LCD shutter glasses synchronized to the display monitor.

– Two basic 3D free-form shapes were created. The simple one was represented by a medusa-like object, the complex shape was a terrain.

Experimental Method

Results

Results

Findings

• Subjects estimated the depth of 3D objects more accurately when objects were displayed with binocular disparity than when only monocular cues were provided.

• The effect of object complexity was highly significant. Complex objects depth was underestimated while simple objects depth was estimated more accurately.