Multimedia Systems &Interfaces

Karrie G. KarahaliosSpring 2007

Perception

• Review

• Expectations

• Perception

• Homework

Color and Visual System

• Color refers to how we perceive a narrow band of electromagnetic energy– source, object,

observer

• Visual system transforms light energy into sensory experience of sight

Human Visual System

• Eyes, optic nerve, parts of the brain

• Transforms electromagnetic energy

Human Visual System

• Formation– cornea, sclera, pupil,

iris, lens, retina, fovea

• Transduction– retina, rods, and cones

• Processing– optic nerve, brain

Image Formation

• Cornea and sclera• Pupil• Iris• Lens• Retina• Fovea Cornea

Sclera

Iris

Len

sPupil

Retina

Fovea

The Cornea

• Part of sclera – hard white part of the eye

• Transparent part at front of eye

• Allows light to enter, refraction occurs

Cornea

Sclera

The Pupil and Iris

• Controls amount of light passing through– diameter varies in

response to light

• Iris controls the diameter of the pupil– gives eye its color

Pupil

Iris

The Lens

• Focuses light on the retina using refraction

• Changes shape to provide focus– spherical for

closer objects– flat for far objects– accommodation

Len

s

The Retina and Fovea

• Retina has photosensitive receptors at back of eye

• Fovea is small, dense region of receptors

– only cones (no rods)

– gives visual acuity

• Outside fovea

– fewer receptors overall

– larger proportion of rods

Fovea

Retina

The Transduction

• Transform light to neural impulses

• Receptors signal bipolar cells

• Bipolar cells signal ganglion cells

• Axons in the ganglion cells form optic nerve

RodsBipolar cellsConesGanglion

Optic nerve

Rods and Cones

• Contain photo-pigment• Respond to low energy• Enhance sensitivity• Concentrated in retina,

but outside of fovea• One type, sensitive to

grayscale changes

• Contain photo-pigment• Respond to high energy• Enhance perception• Concentrated in fovea,

exist sparsely in retina• Three types, sensitive to

different wavelengths

Cones Rods

Rod and Cone Destiny

From http://hyperphysics.phy-astr.gsu.edu/hbase/vision/rodcone.html

120 million rods

6-7 million cones

Tri-stimulus Theory

• 3 types of cones (6 to 7 million of them)– Red (64%), Green (32%), Blue (2%)

• Each type most responsive to a narrow band– red and green absorb most energy, blue the least

• Light stimulates each set of cones differently, and the ratios produce sensation of color

Tri-stimulus Theory

Opponent-Color Theory

• Visual system contains two types of color-sensitive units– red / green; blue / yellow

• Each component in a unit responds opposite the other component– e.g., if red-green responds to red, then

green is inhibited

• Explains concept of ‘after images’

Visual System Facts

• Distinguish hundreds of thousands of colors– more sensitive to brightness

• Can distinguish about 28 fully saturated hues– less sensitive to hue changes in less saturated colors

• Can distinguish about 23 levels of saturation for fixed hue and lightness

• 10 times less sensitive to blue than red or green– it absorbs less energy in the blue range

Physical Properties of Color

• Dominant wavelength– electromagnetic waves– 400nm (violet) to 700nm (red)

• Excitation purity

• Luminance

Spectral Distribution

• Dominant wavelength– spike in power (e2)– white light is uniform

energy distribution

• Excitation purity– ratio between e2 / e1E

nerg

y di

strib

utio

n

Wavelength

e1

e2

Perceptual Properties of Color

• Hue– distinguishes named colors, e.g., RGB– dominant wavelength of the light

• Saturation– how far color is from a gray of equal intensity

• Brightness (lightness)– perceived intensity

Color Perception

• Hue– distinguishes named colors, e.g., RGB– dominant wavelength of the light

• Saturation– how far color is from a gray of equal intensity

• Brightness (lightness)– perceived intensity

White

Grays

Black

Tones

Shades

Tints Purecolors

CIE

• Standard reference

Color Models

Subtractive (CMY)Additive (RGB)

Applies to light-emittingsources (TVs, monitors, etc)

Applies to reflected light

(printed images, paints, etc)

HSV (aka HSB)

• User-oriented, based on use of tints, shades, and tones

RGB

• Based on the fact that the human visual system maintains three types of cones (RGB cones)

• Different weightings produce different colors

Black(0,0,0)

Red(1,0,0)

White(1,1,1)

Green(0,1,0)

Cyan(0,1,1)

Magenta(1,0,1)

Yellow(1,1, 0)

Blue(0,0,1)

YIQ (aka YUV)

• Used in US television broadcasting

• Recoding of RGB for efficiency and compatibility with black-and-white TV

• Y is luminance (not yellow!)

• I and Q is chromaticity

B

G

R

Q

I

Y

311.523.212.

321.275.596.

114.587.299.=

CMYK

• Subtractive color model– used for printing, painting, etc.

• CMY are the complements of RGB– two complementary colors gives a primary

B

G

R

Y

M

C

=

1

1

1_

Gestalt

Auditory Perception

Waves

• Periodic disturbance that travels through a medium (e.g. air or water)

• Transport energy

• Transverse or longitudinal

• Electromechanical or mechanical

Sound

• A longitudinal, mechanical wave– caused by a vibrating source

• Pack molecules at different densities– cause small changes in pressure

• Model pressure differences as sine waves

Volume and Pressure

Auditory System

• Ears, parts of brain, and neural pathways

• Changes in pressure move hair-like fibers within the inner ear

• Movements result in electrical impulses sent to the brain

Process of Hearing (Transduction)

Frequency (temporal) Theory

• Periodic stimulation of membrane matches frequency of sound– one electrical impulse at every peak– maps time differences of pulses to pitch

• Firing rate of neurons far below frequencies that a person can hear– Volley theory: groups of neurons fire in well-

coordinated sequence

Place Theory

• Waves move down basilar membrane– stimulation increases, peaks, and quickly tapers

– location of peak depends on frequency of the sound, lower frequencies being further away

Physical Dimensions

• Amplitude

– height of a cycle

– relates to loudness

• Wavelength (w)

– distance between peaks

• Frequency ( )

– cycles per second

– relates to pitch w = velocity

• Most sounds mix many frequencies & amplitudes

Sound is repetitive changesin air pressure over time

Psychological Dimensions

• Loudness– higher amplitude results in louder sounds– measured in decibels (db), 0 db represents

hearing threshold

• Pitch– higher frequencies perceived as higher pitch– hear sounds in 20 Hz to 20,000 Hz range

Psychological Dimensions

• Timbre (tam-bre)– complex patterns added to the

lowest, or fundamental, frequency of a sound, referred to as spectra

– spectra enable us to distinguish musical instruments

• Multiples of fundamental frequency give music

• Multiples of unrelated frequencies give noise

Sound Intensity

• Intensity (I) of a wave is the rate at which sound energy flows through a unit area (A) perpendicular to the direction of travel

P measured in watts (W), A measured in m2

• Threshold of hearing is at 10-12 W/m2

• Threshold of pain is at 1 W/m2

A

P

t

E

AI

1

Decibel Scale

• Describes intensity relative to threshold of hearing based on multiples of 10

0log10I

IdB

Decibels of Everyday Sounds

Sound Decibels

Rustling leaves 10

Whisper 30

Ambient office noise 45

Conversation 60

Auto traffic 80

Concert 120

Jet motor 140

Spacecraft launch 180

Loudness from Multiple Sources

• Use energy combination equation

where L1, L2, …, Ln are in dB

)10...1010log(10 1010

2

10

1 LNLL

L

Exercises

• Show that the threshold of hearing is at 0 dB• Show that the threshold of pain is at 120 dB

• Suppose an electric fan produces an intensity of 40 dB. How many times more intense is the sound of a conversation if it produces an intensity of 60 dB?

• One guitar produces 45 dB while another produces 50 dB. What is the dB reading when both are played?

• If you double the physical intensity of a sound, how many more decibels is the resulting sound?

Loudness and Pitch

• More sensitive to loudness at mid frequencies than at other frequencies– intermediate frequencies at [500hz, 5000hz]

• Perceived loudness of a sound changes based on the frequency of that sound– basilar membrane reacts more to intermediate

frequencies than other frequencies

Fletcher-Munson Contours

Each contour represents an equal perceived sound

Masking

• Perception of one sound interferes with another

• Frequency masking

• Temporal masking

Frequency Masking

• Louder, lower frequency sounds tend to mask weaker, higher frequency sounds

From http://www.cs.sfu.ca/CourseCentral/365/

Frequency Masking

• Louder, lower frequency sounds tend to mask weaker, higher frequency sounds

Frequency Masking

• Louder, lower frequency sounds tend to mask weaker, higher frequency sounds

Temporal Masking

• When exposed to a loud sound, the human ear contracts slightly to protect delicate structures

• Causes louder sounds to overpower weaker sounds just before and just after it

From http://www.cs.sfu.ca/CourseCentral/365/

Combined Masking

From http://www.cs.sfu.ca/CourseCentral/365/

Localization of Sound

• Localization occurs because– sound reaches one ear before the other– the head creates a ‘sound shadow’ that

decreases intensity of the sound to far ear

Assignment

• Read pamphlets by Andries Van Dam:

Color

Illumination

Intensity Demo

Density Demonstration