Early Perceptual Processing, Part I

The Visual System

THE RETINA

The vertebrate retina has ten distinct layers1. Inner limiting membrane

2. Nerve fiber layer

3. Ganglion cell layer: Layer that contains nuclei of ganglion cells and gives rise to optic nerve fibers

4. Inner plexiform layer

5. Inner nuclear layer

6. Outer plexiform layer : In the macular region, this is known as the Fiber layer of Henle

7. Outer nuclear layer

8. External limiting membrane : Layer that separates the inner segment portions of the photoreceptors

from their cell nuclei

9. Photoreceptor layer: Rods / Cones

10. Retinal pigment epithelium

NEURAL PROCESSING AND EARLY VISION Each retina contains 126 million receptors

2 kinds 120 million Rods

6 million Cones

RECEPTOR CELLS Humans have two receptor cell types:

All receptor cells have outer segments (containing the photoreactive compounds necessary for converting light energy into chemical energy), an inner segment, a perikaryal region, and a terminal

Rods will have long outer segments and small, spherically shaped terminals (spherules)

Cones often have a shorter outer segment, a fatter inner segment and a larger terminal (pedicles) than rods Note: In the human, foveal

cones appear much more like rods than the parafoveal and peripheral cones

CELLULAR ORGANIZATION

All vertebrate retinas are organized according the same basic plan (i.e., two synaptic layers)

Outer & Inner nuclear layers: contains photoreceptors and preprocessing neural elements

Ganglion cell layer: Contains some preprocessing neural elements and the ganglion cells

CELLULAR ORGANIZATION

Although only the photoreceptors are the light sensitive component (all visual responses are initiated by the photoreceptors), the retina has five other basic classes of retinal neurons:

Horizontal Cells

Amarcrine Cells

Bipolar Cells

Innerplexiform Cells

Ganglion Cells

CELLULAR ORGANIZATION The cell bodies (perikarya) of

the photoreceptors are located in the outer nuclear layer

The perikarya of the Horizontal cells lie along the outer margin of the inner nuclear layer

Bipolar perikarya are predominantly located in the middle layer

Amacrine & Innerplexiform cell perikarya are located along the proximal border of the inner nuclear layer

Ganglion cell perikarya comprise the most proximal layer, the ganglion cell layer

SPATIAL SUMMATION

SPATIAL SUMMATION

126 million photoreceptors, but only 1 million ganglion cells

In some parts of the eye a single ganglion cell is receiving inputs from many photoreceptors

This has implications for visual sensitivity and acuity

SPATIAL SUMMATION

The degree of convergence among photoreceptors onto ganglion cells increases with retinal eccentricity (moving from the center out into the periphery)

Foveal cones have a 1:1 or near 1:1 relation with ganglion cells

Receptors in the far periphery can have up to a 400:1 relation with a ganglion cell

SPATIAL SUMMATION

The concept of the Receptive Field A region of space (visual, auditory,

somatosensory) that is associated with a particular responseIn this case, the “region of space” is a section of retina, and the “response” is ganglion cell activity

As the degree of convergence increases, so does the size of the receptive field

SPATIAL SUMMATION Implications for visual sensitivity

With a larger receptive field, more light can be caughtWith more photoreceptors active and pooling their collective responses onto a single ganglion cell, it is much more likely for the ganglion cell to fire

With a smaller receptive field, less light can be caughtWith fewer photoreceptors active, it is less likely for the ganglion cell to fire

SPATIAL SUMMATION

Implications for visual acuityLarge receptive fields pool light

information from a large area Small details get lost It does not matter where in the receptive field

light has fallen, only that enough of it has

The fovea has very small receptive fields, and good acuity

The peripheral retina has larger receptive fields, and poor acuity

RECEPTIVE FIELDS & INHIBITION

Recall the concepts of excitation and inhibition Excitation = more activity Inhibition = less activity

Within a receptive field, some photoreceptors are excitatory, whereas others are inhibitory to the ganglion cell

LATERAL ANTAGONISM

Inhibitory-center-excitatory-surround

Excitatory-center-Inhibitory-surround

CENTER-SURROUND ANTAGONISM

Lateral AntagonismAs you increase the size of the spot of light on

the entire receptive field, the firing rate of the ganglion cell changes

RECEPTIVE FIELDS & INHIBITION

Implications for visual perception Mach bands

The perception of light and dark bands near the borders between light and dark areas

MACH BANDS

WHAT IS THE BENEFIT? Although LA can lead to some interesting visual

illusions, it does serve very important functions

E.g. Contrast enhancement at edges

Your eye is not a perfect optical instrument

Edges are blurred to some degree

By enhancing the difference between a light and dark region, your visual system is able to compensate for the blurring

Physical

Optical

Perceived

Inte

nsity

Inte

nsity

Inte

nsity

Space

Space

Space

THE OPTIC NERVE

Neurons leave the eye via the optic nerve

Highly myelinated and therefore neural conduction is very fast

Diseases that affect the myelination of neurons exhibit their symptoms here first (e.g., multiple sclerosis)

THE PATHWAY TO THE BRAIN After leaving the eye…

The optic nerve crosses at optic chiasm

Optical fibers extend away from the optic chiasm Travel to lateral geniculate nucleus (dLGN)

Optical radiation extends away from the dLGN branch out and enter the primary visual cortex AKA striate cortex

Signals are then rerouted to higher brain areas (extra-striate cortex)

OPTIC CHIASM

Why the crossover?

Contralateral processing

Right visual field is processed in the left hemisphere of the brain, and vice versa

THE LATERAL GENICULATE NUCLEUS

The dLGN has circular receptive fields with a center-surround configuration (more on this later)

Retinotopic map Adjacent points on the

retina are adjacent points at the dLGN

THE LATERAL GENICULATE NUCLEUS Organized into 6 layers

the eye on the same side sends information to layers 2, 3 and 5 (ipsilateral)

the eye on the opposite side sends information to layers 1, 4 and 6 (contralateral)

Just remember that 2 + 3 = 5 whereas 1 + 4 does not equal 6

THE PRIMARY VISUAL CORTEX

Contralateral processing The LEFT hemisphere processes information from

the RIGHT visual field

The RIGHT hemisphere processes information from the LEFT visual field

Retinotopic map Cortical (foveal) magnification

Fovea is only 0.01% of retinal area, but 8-10% of the cortical area (1000x magnification)

THE PRIMARY VISUAL CORTEX

The visual cortex is topographically organized

A large area of the visual cortex is mapped to the central portion of the visual field (retina)

Retinotopic map & Cortical magnification

Number of neuronal cells in cerebral cortex

neurons ----------- 10-15 billion

glial cells ---------- 50 billion

CEREBRAL CORTEX

CEREBRAL CORTEX

1. Pyramidal Cell

2. Fusiform Cell

3. Granular (Stellate) Cell

4. Basket cell

5. Double bouquet cell

6. Chandlier cell

7. Neurogliform cell

8. Horizontal Cell of Cajal

9. Cells of Martinotti

10. Axon

I. Molecular Layer

II. External Granular Layer

III. External Pyramidal Layer

IV. Internal Granular LayerI. Line of Gennari

in area 17

V. Internal Pyramidal Layer

VI. Polymorphic Layer

Golgi Nissl Weigert

THE “WHAT” PATHWAY

Information leaving the striate cortex and entering the temporal lobe is associated with

object recognition

Ventral pathway

Ablating or damaging this pathway results in impairments of visual object recognition E.g. Agnosia

THE “WHERE” & “HOW” PATHWAY

Information leaving the striate cortex and entering the parietal lobe is associated with object location (spatial processing)

Dorsal pathway

Ablating or damaging this pathway results in impairments of visual object localization

THE ORIGINS OF THESE PATHWAYS

Dorsal and ventral pathway separation begins at the level of the retina M-cells

P-cells

THE MAGNOCELLULAR PATHWAY

M-cells

Retinal ganglion cells with large cell bodies

Innervate layers 1 and 2 of the dLGN

Fed mainly by rods (rod dominated)

Associated mainly with motion perception

Signals in fast and transient bursts

THE PARVOCELLULAR PATHWAY

P-cells

Retinal ganglion cells with small cell bodies

Innervate layers 3, 4, 5 and 6 of the dLGN

Fed mainly by cones (cone dominated)

Associated mainly with color, texture and depth perception

Signals in slow and sustained fashion

Magnocellular(M-cell)

Parvocellular(P-cell)

LGN layers

1 and 2

LGNlayers3 - 6

V1Layer 4Cα

Parietal Lobe

Temporal LobeV1

Layer 4Cβ

Retina BrainDorsal Pathway

Ventral Pathway

PARVO & MAGNO PATHWAYS

In primates, it is clear that the retinocortical visual system is organized into two major pathways (parvo & magno)

The neurons which comprise these two pathways have different sensitivities to chromatic, spatial and temporal stimuli

PARVO & MAGNO PATHWAYS

Characteristics of parvo and magno neurons located in the primate retina and dLGN

Characteristics Parvo Neurons Magno Neurons

Color coding Color opponent Weak or no color opponency

Temporal responsiveness

Sustained Transient

Speed of transmission Slow Fast

Spatial linearity Linear Linear or nonlinear

Retinal distribution Central Peripheral

Spatial sensitivity High frequencies Low frequencies

Response to increasing contrast

Weak Saturates

Cortical projection (V1)

4A, 4Cβ 4Cα

SUSTAINED VS. TRANSIENT RESPONSE

Sustained

Transient

FUNCTIONS OF THE M & P PATHWAYS

Behavioral studies in monkeys Legion of the parvocellular region of the dLGN:

Color vision is severely diminishedHigh frequency spatial vision is poor Interestingly, high frequency flicker

detection remains largely unaffected

Legion of the magnocellular region of the dLGN:Color vision is largely retainedNormal contrast sensitivity to high spatial

frequenciesProfound loss of sensitivity to high

frequency flicker

RETINOTOPIC MAP

THE STRIATE CORTEX

Named so because of the dense plexus of geniculate axons that form distinctive bands The “primary” band is

referred to as the line of Gennari (layer 4B)

THE STRIATE CORTEX

THE STRIATE CORTEX

Conventionally divided into 6 layersLayer 1 is near the

cortical surface; layer 6 is adjacent to the white matter

THE STRIATE CORTEX

The primary destination of the axons from the LGN is to layer 4, although some fibers will extend to layers 1, 3 and 6

THE STRIATE CORTEX

Organization might be very similar to the retina and LGN

Evidence for presence of horizontal (lateral) connections between areas of V1

Physiological evidence suggests that they are largely inhibitory

THE STRIATE CORTEX

The striate cortex sends axon projections to the extrastriate cortex This region of the visual cortex is not

distinguished by a line of Gennari

Extrastriate areas include: Visual Area 2 (V2) Visual Area 4 (V4) Inferotemporal Cortex (IT) Mediotemporal Cortex (MT; V5)

THE “FLOW” OF INFORMATION

Recall that the magno and parvo-cellular systems have been considered largely distinct This is true, however, at higher cortical areas

more and more information begins to “cross-over”

Obviously, this is important for a complete sensory integration, or else we could see colors, but never associate them with a particular object, per se

THE “FLOW” OF INFORMATION

Not only does the striate cortex project “forward” through the visual system, but it also projects “backward”

Reciprocal projects are sent back to the dLGN and superior colliculus

THE “FLOW” OF INFORMATION

These “backward” projections originate from the deeper layers of the cortex I.e., Layers 5 & 6

“Forward” projections tend to originate from the superficial layers

THE “FLOW” OF INFORMATION

Interestingly, information can then flow back to the cortex forming a feedback loop

The ratio of feed-back to feed-forward projections can be as high as 900:1!

THE STRIATE CORTEX

Very little was understood about the primary visual cortex Although it had been known that concentrically

organized cells could be found at the level of the retina, these structures could not be found in V1

Stimuli optimized for a circular configuration were not “activating” cells in the visual cortex…

THE STRIATE CORTEX

Hubel & Wiesel (1959; 1962) made a critical discovery when making extracellular recordings from single cells in the cat striate cortex Essentially, they discovered neurons that were

most sensitive to elongated stimuli such as bars or edges

These cells were then divided into two general categories: Simple cells

Complex Cells

Watch the video!

CORTICAL CELLS

Simple cellsRespond best to a bar of light in a particular

orientation

Complex cellsRespond best to a bar of light of a particular

orientation moving in a particular direction

End-stop cellsRespond best to bars of a specific length, or to

joined lines (corners) moving in a specific direction

THE STRIATE CORTEX Neurons are specialized to respond best to

very specific aspects of a stimulus Orientation, size, direction of movement, etc.

Arranged in a side-by-side fashion, not center-surround fashion Will respond best to bars of light that share the

same orientation as the receptive field

SIMPLE CELLS

It is thought that that simple cell receptive fields are formed by the addition of dLGN receptive fields that lie along a straight line

This is known as hierarchical or serial processing

COMPLEX CELLS

Like the simple cells, complex cells respond best to elongated stimuli of a particular orientation However, unlike simple cells, the object can be

located anywhere within the receptive boundaries of the field

COMPLEX CELLS

Moreover, many complex cells are characterized by a sensitivity to a stimulus moving in a particular direction A stimulus will all the proper characteristics, but

moving in the direction opposite to that which the cell is sensitive will illicit no response

Complex cells also cannot be divided into discrete excitatory or inhibitory regionsSuggests that the integration of multiple simple cells is a non-linear process

END-STOP CELLS

Originally classified as hypercomplex cells Sensitive to the length of an edge or bar stimulus

It was later determined that this is a characteristic of many of the earlier cells, and so does not define a new class of cell

Nevertheless, Hubel & Wiesel’s work conclusively demonstrated that hierarchical processing is central to processing of visual information As information is conveyed to higher and higher

vision centers, the neurons become increasingly more specific in their response

STRIATE CORTICAL ARCHITECTURE

Many cortical neurons are binocular (receive input from both eyes)

Binocular cells may mediate stereopsis The receptive fields of

many binocular cells do not overlap with the sameregions in the eye

Permits the coding of retinal disparity

STRIATE CORTICAL ARCHITECTURE

Many cortical neurons are binocular (receive input from both eyes)

Binocular cells may mediate stereopsis The receptive fields do

overlap when an object islocated at a critical distancefrom the eyes

STRIATE CORTICAL ARCHITECTURE

Ocular dominance Columns Ocular dominance is laid out

in a regular pattern of alternating right and left ocular dominance slabs

A complete set of ocular dominance columns and orientation columns forms a hypercolumn (2x2mm)

PARVO AND MAGNO PATHWAYS

Recall that the segregation of the parvo and magno systems persists through to V1, and somewhat beyond

Staining the cortex for cytochrome oxidase reveals an irregular pattern of “blobs” within the superficial layers of the striate cortex Stripes adjoining area V2 can also be seen

PARVO AND MAGNO PATHWAYS Blobs are rich with concentrically organized, double-

color opponent neurons

Blobs are also connected to the stripes in V2 and this constitutes a continuation of the color sensitive parvo pathway

The superficial layers between the blobs are cleverly known as the interblob region

PARVO AND MAGNO PATHWAYS

The magno pathway appears to bypass the blob and interblob regions by synapsing with deeper layers of the cortex The magno pathway projects to the stripes in V2

and then to V5 (motion)

CORTICAL MODULARITY

Evidence that specific brain regions respond best to particular classes of stimuli comes from legion and imaging studies With regard to brain imaging studies:

When subjects are presented with particular kinds of visual stimulation, it is found that distinct regions of the brain become “active” and/or inhibited

CORTICAL MODULARITY

Higher visual areas appear to have specialized functions Visual Area 4 (V4): responds to chromatic stimuli

and is therefore involved in color perception

Inferotemporal Cortex (IT): responds to complex forms (e.g., faces), indicating a role in form perception

Both of these brain regions are considered to be a part of the parvo (“What”) stream

CORTICAL MODULARITY

Higher visual areas appear to have specialized functions Visual Area 5 (V5): responds to motion stimuli

Therefore, this brain region is associated with the magno (“Where”) stream

V3 processes Dynamic form

V4 processes color and form with color

Cerebral achromatopsia: color blindness due to damage to V4

V5 processes motion

VISUAL AREA 5

This region of the brain is heavily studied and is probably better understood than other regions Neurons in V5 respond to global movement

Motion Aftereffect

Early Perceptual Processing, Part II

The Auditory System

WHY IS HEARING IMPORTANT

The first sense to develop

The first sound you hear is your mother’s heart beat Infants react to

brady-tachycardia Used in

poems/music, iambic pentameter(Wagner’s 9th

symphony)

WHY IS HEARING IMPORTANT Approximately 2 million people are profoundly deaf

One of every 1000 infants is born totally deaf

28 million Americans have hearing loss

80 percent of those affected have irreversible and permanent hearing damage

Average age of diagnosis, 3 years of age

15 percent of the U.S. population is affected by tinnitus

Presbycusis affects 1/3 of the U.S. population over 65

Estimated cost of care, 56 billion per year

“PARENTESE”

Middle Ear

Tympanic membrane

malleus

Ossicles (middle-ear bones)

Tensor tympani and stapedius muscles dampen ossicular transmission of loud sounds

Stapes to theOval window

SOUND

Physical definition Sound is rhythmic pressure changes in air or

some other medium

Perceptual definition Sound is the experience we have when we hear

something

THE COCHLEA

The cochlea is the organ of the inner ear responsible for transducing the auditory stimulus into electrochemical signals

Small fluid filled bony structure rolled upon itself (2¾ turns)

THE COCHLEA The base is the portion of the cochlear

nearest to the oval window, whereas the apex is the area furthest from the oval window

Divided into two halves by the cochlear partition which extends the length of the inner cochleaTop half: scala vestibuliBottom half: scala tympani

On the cochlear partition is a structure called the organ of Corti

THE COCHLEA UNCOILED

Base

Oval Window

Apex

THE ORGAN OF CORTI Supported by the Basilar Membrane

Contains the hair cells which are the receptors for hearing

Cilia protrude from the tops of the cells and are responsible for converting vibrations into electrical signals Outer hair cells Inner hair cells

The tectorial membrane extends over the hair cells

AUDITORY TRANSDUCTION Transduction starts with the bending of the

cilia What causes the bending?

The in-and-out movements of the stapes on the oval window generates pressure changes in the fluid within the cochlea

The rhythmic pressure changes cause the cochlear partition to vibrate up-and-down

AUDITORY TRANSDUCTION

Up-and-down motion of the partitionCauses the Organ of Corti to vibrate

Causes the tectorial membrane to move back-and-forth

These two motions cause the cilia of the inner hair cells to bend because of their movement against the surrounding liquid and because they are in contact with the tectorial membrane

AUDITORY TRANSDUCTION

The necessary amount of bending of the cilia to generate an electrical signal is VERY small

100 picometers

To put this in scale, if you increased the size of a cilium to be the same as the Eiffel Tower, the minimal displacement would be 10mm

Efferent

Afferent

Outer Hair Cells

(OHC)

Inner Hair Cells (IHC)

Auditory Transduction

Depolarize: Influx of Ca++Hyperpolarize: Efflux of K+

THE COCHLEA

IHCs send signals to fibers which bundle together and become the auditory nerve95% of the auditory nerve caries signals

from the IHCs

OHCs also send signals through the auditory nerve5% of the auditory nerve caries signals

from the OHCs

BÉKÉSY’S PLACE THEORY Tonotopic Map

An orderly map of frequencies along the length of the cochlea

Placing electrodes at different places along the length of the cochlea and measuring the electrical response to different frequencies of sound indicates that some areas respond best to low freqs, and other areas to high freqs

Specifically, the base of the cochlea responds best to high freqs, and the apex responds best to low freqs

THE MOTILE RESPONSE Although the inner hair cells are responsible

for transducing the auditory stimulus into electrical signals, the outer hair cells also play an important role

Movement of the outer hair cells affects the movement of the basilar membrane Different outer hair cells will respond to different

frequencies (high freq = base, etc.) Movement of the outer hair cells amplifies the

motion of the basilar membrane and sharpens its response to specific frequencies

Ultimately, this helps the ear distinguish between two very similar tones (ex: 400 vs. 405Hz)

AUDITORY PATHWAY Fibers from the

cochlea bind together and form the auditory nerve

The auditory nerve then travels to the cochlear nucleus and then the superior olivary nucleus of the brain stem

From here, signals are sent to the inferior colliculus in the midbrain and then…

AUDITORY PATHWAY To the Medial

geniculate nucleus (MGN) in the thalamus

From the Thalamus (MGN) signals are send to the primary auditory cortex (A1) in the temporal lobe

SONIC MG Superior Olivary

Nucleus; Inferior Colliculus; Medial Geniculate Nucleus

THE PRIMARY AUDITORY CORTEX (A1) Tonotopic organization

Frequency analysis (perceiving pitch)

Binaural integration Auditory localization

THE PRIMARY AUDITORY CORTEX (A1)

FROM A1…

The auditory system (like the visual system) has What and Where pathwaysWhat stream

Auditory ventral pathway Passes to the pre-frontal cortex Associated with identifying sounds

Where stream Auditory dorsal pathway Passes to the parietal cortex Associated with identifying the location of sounds

AUDITORY LOCALIZATION

Coordinate system Azimuth – horizontal (left to right / side to side)

Elevation – vertical (top to bottom / up-down)

Distance – how far away is the sound

AUDITORY LOCALIZATION

People with normal hearing tend to be fairly good at identifying the location of sounds Listeners can localize sounds from directly in

front of them most accurately (within 2 - 3.5 deg)

Listeners are least accurate localizing sounds that are off to the side or behind their head (error = 20+ deg)

AUDITORY LOCALIZATION

Binaural Cue Auditory cue based on sounds reaching both ears

Monaural Cue Auditory cue based on sound only reaching one

ear

BINAURAL CUES

Based on a comparison of sound signals reaching the left and right ears Sounds to the side of a listener will…

…reach one ear sooner than the other Interaural time differences (ITD)

…be more intense in one ear than the otherInteraural intensity differences (ILD)

Primarily involved in azimuth judgments

Down to 10 microseconds

INTERAURAL TIME DIFFERENCE

INTERAURAL INTENSITY DIFFERENCE

THE ACOUSTIC SHADOW

Acoustic Shadow

COMPUTATION OF SPATIAL LOCALIZATION

The brainstem computes spatial location of sounds by using Delay Lines and cells called Coincidence Detectors

These structures are able to detect the difference in arrival time of a sound to each ear Certain cells in the brainstem respond

preferentially to particular timing discrepancies between the two ears

COMPUTATION OF SPATIAL LOCALIZATION

A B C

Left Cochlear Nucleus

Right Cochlear Nucleus

Delay Line

Medial SuperiorOlivary Nucleus

(MSO)

Cells A, B & C are Coincidence Detectors: Activated most strongly when signals from BOTH cochlear nuclei arrive at the same time.

ACOUSTICAL ILLUSIONS

The McGurk Effect Our perception of speech is multimodal, that is, it

can be affected by many different senses

The McGurk Effect illustrates how visual information can affect auditory perception

Ss hear / ba-ba /, but see a person saying / ga-ga /

The resulting perception is / da-da /

McGurk Link