Connectionist models and their properties.pdf

8/11/2019 Connectionist models and their properties.pdf

1/50

CO GN IT IVE SCIENCE 6 , 205 -25 4 1982)

C nnect i n ist M od els and T h ei r Prop erties

J A F E L D M A N A N D D H B A L L A R D

C o m p u t e r S c ie n c e D e p a r t m e n t

Univers i t y o f Roches ter

Roches ter N Y 14627

M uch o f t he p rogress in the f i e lds const it u ting cogn i t ive sc ience has be en ba sed

upon t he use o f exp l ic i t in f o rm at ion p rocess ing m odels a lmo st exc lus ive ly

p a t t e r n e d a f t e r c o n v e n t i o n a l s e r i a l c o m p u t e r s . A n e x t e n s i o n o f th e s e i d e a s t o

m a s s iv e ly p a r a l l e l c o n n e c t i a n i s t m o d e l s a p p e a r s t o o f f e r a n u m b e r o f a d v a n -

t ages . A f t e r a p re l imina ry d iscuss ion t h is pap er in t roduces a ge ne ra l connec-

t ion is t m od e l and cons iders h ow i t m igh t be used in cog n i t i ve sc ience . Am on g

t he issues add resse d are : s t ab i l i ty and no ise -sens it iv it y d is t r ibu t ed dec is ion -

m a k i n g t im e a n d s e q u e n e p r o b le m s a n d t h e re p r e s e n t a t io n o f c o m p l e x

concepts.

1 . I N T R O D U C T I O N

M u c h o f t h e p r o g r e s s i n t h e f i e l d s c o n s t i t u t i n g c o g n i t i v e s c i e n c e h a s b e e n

b a s ed u p o n t h e u s e o f c o n c r e t e i n f o r m a t i o n p r o c e ss i n g m o d e l s I P M ) ,

a l m o s t e x c lu s iv e ly p a t t e r n e d a f t e r c o n v e n t i o n a l s e q u e n t i a l c o m p u t e r s .

T h e r e a r e s ev e ra l r e a s o n s f o r t ry i n g t o e x t e n d I P M t o c a se s w h e r e th e c o m -

p u t a t i o n s a r e c ar r ie d o u t b y a p a r a ll e l c o m p u t a t i o n a l e n g i n e w i th p e r h a p s

b i l l i o n s o f a c t i v e u n i t s . A s a n i n t r o d u c t i o n , w e w i l l a t t e m p t t o m o t i v a t e t h e

c u r r e n t i n t e r e s t i n m a s s i v e l y p a r a l l e l m o d e l s f r o m f o u r d i f f e r e n t p e r s p e c -

t iv e s: a n a t o m y , c o m p u t a t i o n a l c o m p l e x i t y , t e c h n o l o g y , a n d t h e r o l e o f f o r -

m a l l a n g u a g e s i n s c i e n c e . I t is t h e l a s t o f t h e s e w h i c h is o f p r i m a r y c o n c e r n

h e re . W e w ill fo c u s u p o n a p a r t i c u l a r f o r m a l i s m , c o n n e c t i o n i s t m o d e l s

C M ) , w h i c h is b a s e d e x p l ic i tl y o n a n a b s t r a c t i o n o f o u r c u r r e n t u n d e r s t a n d -

i n g o f t h e i n f o r m a t i o n p r o c e s s i n g p r o p e r t i e s o f n e u r o n s .

A n i m a l b r a in s d o n o t c o m p u t e l ik e a c o n v e n t i o n a l c o m p u t e r . C o m -

p a r a t i v e l y s lo w m i l l is e c o n d) n e u r a l c o m p u t i n g e l e m e n t s w i t h c o m p l e x ,

p a r a ll e l c o n n e c t i o n s f o r m a s t ru c t u r e w h i c h is d r a m a t i c a l l y d i f f e r e n t f r o m a

h i g h -s p e e d , p r e d o m i n a n t l y s er ia l m a c h i n e . M u c h o f c u r r e n t r e s e a rc h in t h e

n e u r o s c i e n c e s is c o n c e r n e d w i t h t r a c in g o u t t h e se c o n n e c t i o n s a n d w i t h d is -

c o v e ri n g h o w t h e y t r a n s f e r i n f o r m a t i o n . O n e p u r p o s e o f t h i s p a p e r is t o

s u g g e s t h o w c o n n e c t i o n i s t t h e o r i e s o f t h e b r a i n c a n b e u s e d t o p r o d u c e

2 0 5


2/50

2 6 F E L D M A N A N D B A LL AR D

t e s t a b l e , d e t a i l e d m o d e l s o f i n t e r e s t i n g b e h a v i o r s . T h e d i s t r i b u t e d n a t u r e o f

i n f o r m a t i o n p r o c e s si n g in t h e b r a i n is n o t a n e w d i s c o v e r y . T h e t r a d i t i o n a l

v ie w (w h i c h w e s h a r e d ) is t h a t c o n v e n t i o n a l c o m p u t e r s a n d l a n g u a g e s w e r e

T u r i n g u n i v e rs a l a n d c o u l d b e m a d e t o s i m u l a t e a n y p a r a l le l is m ( o r a n a l o g

v a lu e s ) w h i c h m i g h t b e r e q u i r e d . C o n t e m p o r a r y c o m p u t e r s c ie n c e h a s s h a r p -

e n e d o u r n o t io n s o f w h a t is c o m p u t a b l e t o i n cl ud e b o u n d s o n t im e , s t o r-

a g e , a n d o t h e r r e s o u r c e s . I t d o e s n o t s e e m u n r e a s o n a b l e t o r e q u i r e t h a t

c o m p u t a t i o n a l m o d e l s i n c o g n i t i v e s c i e n c e b e a t l e a s t p l a u s i b l e i n t h e i r

p o s t u l a t e d r e s o u r c e r e q u i r e m e n t s .

T h e c r i ti c a l r e s o u r c e th a t is m o s t o b v i o u s is t i m e . N e u r o n s w h o s e b a s i c

c o m p u t a t i o n a l s p e e d i s a f e w m i l l i s e c o n d s m u s t b e m a d e t o a c c o u n t f o r

c o m p l e x b e h a v i o r s w h i c h a r e c a r r ie d o u t i n a f e w h u n d r e d m i l l is e c o n d s

( P o s n e r , 1 9 7 8 ) . T h i s m e a n s t h a t

entire complex behaviors are carried out in

less than a hundred time steps C u r r e n t A I a n d s i m u l a t i o n p r o g r a m s r e q u i r e

m i l l io n s o f t im e st e p s. I t m a y a p p e a r t h a t t h e p r o b l e m p o s e d h e r e is i n h e r -

e n t l y u n s o l v a b l e a n d t h a t t h e r e i s a n e r r o r i n o u r f o r m u l a t i o n . B u t r e c e n t

r e su l ts i n c o m p u t a t i o n a l c o m p l e x i t y t h e o r y ( J a ' J a ' , 1 98 0) s u g ge s t t h a t n e t -

w o r k s o f a c ti v e c o m p u t i n g e l e m e n t s c a n c a r r y o u t a t l e as t s i m p l e c o m p u -

t a t i o n s i n t h e r e q u i r e d t i m e r a n g e . I n s u b s e q u e n t s e c t i o n s w e p r e s e n t f a s t

s o l u ti o n s t o a v a r i e ty o f r e le v a n t c o m p u t i n g p r o b l e m s . T h e s e s o l u t io n s i n -

v o l v e u s in g m a s s i v e n u m b e r s o f u n i ts a n d c o n n e c t i o n s , a n d w e a ls o a d d r e s s

t h e q u e s t i o n s o f l i m i t a t i o n s o n t h e s e r e s o u r c e s .

A n o t h e r r e c e n t d e v e l o p m e n t i s t h e f e a si b il it y o f b u i ld i n g p a r a ll e l c o m -

p u t e r s . T h e r e is c u r r e n t l y t h e c a p a b i l i t y to p r o d u c e c h i p s w i t h 1 0 0 ,0 0 0 g at e s

a t a r e p r o d u c t i o n c o s t o f a f ew c e n ts e a c h , a n d t h e t e c h n o l o g y t o g o to

1 ,0 0 0 ,0 0 0 g a t e s / c h i p a p p e a r s t o b e in h a n d . T h i s h a s tw o i m p o r t a n t c o n s e -

q u e n c e s f o r t h e s t u d y o f C M . T h e o b v i o u s c o n s e q u e n c e is t h a t i t is n o w f e a-

s ib le t o f a b r i c a t e m a s s i v e ly p ar a ll e l c o m p u t e r s , a l t h o u g h n o o n e h a s y e t d o n e

s o ( F a h l m a n , 1 98 0; H i l li s , 1 9 8 1) . T h e s e c o n d c o n s e q u e n c e o f th i s d e v e l o p -

m e n t i s t h e r e n e w e d i n t e r e s t i n t h e b a s i c p r o p e r t i e s o f h i g h l y p a r a l l e l c o m -

p u t a t i o n . A m a j o r r e a s o n w h y th e r e a r e n ' t y e t a n y o f t h e s e C M m a c h i n e s i s

t h a t w e d o n o t y e t k n o w h o w t o d e si g n , a s se m b l e , t e st , o r p r o g r a m s u c h

e n g in e s. A n i m p o r t a n t m o t i v a t i o n f o r t h e c a r e f u l s t u d y o f C M is t h e h o p e

t h a t w e w i ll l e a r n m o r e a b o u t h o w t o d o p a r a l l e l c o m p u t i n g , b u t w e w i ll s a y

n o m o r e a b o u t t h a t i n t h i s p a p e r .

T h e m o s t i m p o r t a n t r e a s o n f o r a se r io u s c o n c e r n i n c o g n it iv e sc ie n c e

f o r C M is t h a t t h e y m i g h t l e a d t o b e t t e r s c i e n c e. I t is o b v i o u s t h a t t h e c h o i c e

o f t ec h n i c a l la n g u a g e t h a t is u se d f o r e x p r e s s in g h y p o t h e s e s h a s a p r o f o u n d

i n f lu e n c e o n t h e f o r m i n w h i ch t h e o r i e s a r e f o r m u l a t e d a n d e x p e r i m e n t s

u n d e r t a k e n . A r t i f i c i a l i n t e ll i g e n c e a n d a r t i c u l a t i n g c o g n i t i v e sc i en c e s h a v e

m a d e g r e a t p r o g r e s s b y e m p l o y i n g m o d e l s b a s e d o n c o n v e n t i o n a l d i g i t a l

c o m p u t e r s a s t h e o r i es o f i n te l li g e nt b e h a v i o r . B u t a n u m b e r o f c r u c ia l

p h e n o m e n a s u c h a s a s s o c i a t i v e m e m o r y , p r i m i n g , p e r c e p t u a l r i v a l r y , a n d


3/50

CO NNE C TI ONI S T MO DE LS A N D THE IR P ROP ERTIE S 2 7

the remarkable recovery ability of animals have not yielded to this treat-

ment. A major goal of this paper is to lay a foundation for the systematic

use of massively parallel connectionist models in the cognitive sciences, even

where these are not yet reducible to physiology or silicon.

Over the past few years, a number of investigators in different fields

have begun to employ highly parallel models (idiosyncratically) in their

work. The general idea has been advocated for animal models by Arbib

(1979) and for cognitive models by Anderson (Anderson et al., 1977) and

Ratcliff (1978). Parallel search of semantic memory and various spreading

act ivat ion theories have become common (though not quite consistent)

parts of information processing modeling. In machine perception research,

massively parallel, cooperative computational theories have become a

dominant paradigm (Marr & Poggio, 1976; Rosenfeld et al., 1976) and

many of our examples come from our own work in this area (Ballard, 1981;

Sabbah, 1981). Scientists looking at performance errors and other non-

repeatable behaviors have not found conventional IPM to be an adequate

framework for their efforts. Norman (1981) has recently summarized argu-

ments from cognitive psychology, and Kinsbourne and Hicks (1979) have

been led to a similar view from a different perspective. It appears to us that

all of these efforts could fit within the CM paradigm outlined here.

One of the most interesting recent studies employing CM techniques is

the partial theory of reading developed in (McClelland & Rumelhart, 1981).

They were concerned with the word superiority effect and related questions

in the perception of printed words, and had a large body of experimental

data to explain. One major finding is that the presence of a printed letter in a

brief display is easier to determine when the letter is presented in the context

of a word than when it is presented alone. The model they developed (cf.

Figure 1) explicitly represents three levels of processing: visual features of

printed letters, letters, and words. The model assumes that there are positive

and negative (circular tipped) connections from visual features to the letters

that they can (respectively, cannot) be part of. The connections between let-

ters and words can go in either direction and embody the constraints of

English. The model assumes that many units can be simultaneously active,

that units form algebraic sums of their inputs and output values propor-

tionally. The activity of a unit is bounded from above and below, has some

memory, and decays with time. All of these features, and several more, are

captured in the abstract unit described in Section 2.

This idea of simultaneously evaluating many hypotheses (here words)

has been successfully used in machine perception for some time (Hanson &

Riseman, 1978). What has occurred to us relatively recently is that this is a

natural mode of computation for widely interconnected networks of active

elements like those envisioned in connectionist models. The generalization

of these ideas to the connectionist view of brain and behavior is that all im-


4/50

208 FELDMAN AN D BALLARD

Figu re 1 . A f e w o f th e n e ig h o r s o f t h e n o d e f o r t h e l e t t e r t i n t he f i rs t p o s i t i o n in a w o r d ,

and the i r i n te rconnec t ions (McC le l land & Rume lha r t , 1981 ) .

portant encodings in the brain are in terms of the relative strengths of

synaptic connections. The fundamental premise of connectionism is that in-

dividual neurons do not transmit large amounts of symbolic information

Instead they compute by being appropriately connected to large numbers of

similar units. This is in sharp contrast to the conventional computer model

of intelligence prevalent in computer science and cognitive psychology.

The fundamental distinction between the conventional and connec-

tionist comput ing models can be conveyed by the following example. When

one sees an apple and says the phrase wormy app le, some information

must be transferred, however indirectly, from the visual system to the

speech system. Either a sequence of special symbols that denote a wormy

apple is transmitted to the speech system, or there are special connections to

the speech command area for the words. Figure 2 is a graphic presentation

of the two alternatives. The path on the right described by double-lined

arrows depicts the situation (as in a computer) where the information that a

wormy apple has been seen is encoded by the visual system and sent as an

abstract message (perhaps frequency-coded) to a general receiver in the

speech system which decodes the message and initiates the appropriate

speech act. Notice that a complex message would presumably have to be

transmitted sequentially on this channel, and that each end would have to


5/50

CONNECTIONISTMODELSAND THEIRPROPERTIES 209

der

~ o d e r

Figure 2 Connectionism s sym bolicencoding

s s u m e s s o m e gen eral encoding

Assumes ndividualconnections

l ea r n th e c o m m o n c o d e f o r e v e r y n e w c o n c e p t . N o o n e h a s y et p r o d u c e d a

b i o l o g ic a l ly a n d c o m p u t a t i o n a l l y p l a u s ib l e r e a l i z at i o n o f t h is c o n v e n t i o n a l

c o m p u t e r m o d e l .

T h e o n l y a lt e r n a t iv e t h a t w e h a v e b e e n a b l e t o u n c o v e r is d e s c r i b ed b y

t h e p a t h w i t h s i n g l e - w i d t h a r r o w s . T h i s s u g g e s t s t h a t t h e r e a r e ( i n d i r e c t )

l i n k s f r o m t h e u n i t s ( c e l l s , c o l u m n s , c e n t e r s , o r w h a t - h a v e - y o u ) t h a t

r e c o g n i z e a n a p p l e t o s o m e u n i ts r e s p o n s i b l e f o r s p e a k in g t h e w o r d . T h e

c o n n e c t i o n i s t m o d e l r e q u i r e s o n l y v e r y s im p l e m e s sa g e s ( e .g . s t i m u l u s

s t r e n g t h ) t o c r o s s a c h a n n e l b u t p u t s s t r o n g d e m a n d s o n t h e a v a i l a b i l i t y o f

t h e r ig h t c o n n e c t i o n s . Q u e s t i o n s c o n c e r n i n g t h e le a r n in g a n d r e i n f o r c e m e n t

o f c o n n e c t i o n s a r e a d d r e s s e d i n F e l d m a n , ( 1 98 1 b ).

F o r a n u m b e r o f r e a s o n s ( i n c l u d i n g r e d u n d a n c y f o r r e l i a b i l i t y ) , i t i s

h i g h ly u n l i k e ly t h a t t h e r e i s e x a c t ly o n e n e u r o n f o r e a c h c o n c e p t , b u t t h e

p o i n t o f v ie w t a k e n h e r e is t h a t t h e a c t i v it y o f a sm a l l n u m b e r o f n e u r o n s

( s ay 10) e n c o d e s a c o n c e p t l i k e a p p l e . A n a l t e r n a t i v e v ie w ( H i n t o n & A n d e r -

s o n , 1 98 1) is t h a t c o n c e p t s a r e r e p r e s e n t e d b y a p a t t e r n o f a c t i v i t y i n a

m u c h l a rg e r s et o f n e u r o n s ( s a y 1 ,0 00 ) w h i c h a l s o r e p r e s e n t m a n y o t h e r c o n -

c e p ts . W e h a v e n o t s e en h o w t o c a r r y o u t a p r o g r a m o f s p e ci fi c m o d e l i n g i n

t e r m s o f t h e se d i f fu s e m o d e l s . O n e o f t h e m a j o r p r o b l e m s w i t h d i f fu s e


6/50

21 FELDMAN AND BALLARD

m o d e l s a s a p a ra l le l c o m p u t a t i o n s c h e m e is c r o s s -t a lk a m o n g c o n c e p t s . F o r

e x a m p l e , i f c o n c e p t s u s i n g u n i t s 1 0 , 2 0 , 3 0 . . . . ) a n d 5 , 15 , 2 5 . . . . ) w e r e

s i m u l t a n e o u s l y a c t i v a t e d , m a n y o t h e r c o n c e p t s , e . g . , 2 0 , 2 5 , 3 0 , 3 5 . . . . )

w o u l d b e a c t i v e a s w e ll . I n t h e e x a m p l e o f F i g u r e 2 , t h is m e a n s t h a t d i f f u s e

m o d e l s w o u l d b e m o r e l i ke t h e s h a r e d s e q u e n t ia l c h a n n e l . A l t h o u g h a si n gl e

c o n c e p t c o u l d b e t r a n s m i t t e d i n p a ra l le l , c o m p l e x c o n c e p t s w o u l d h a v e t o

g o o n e a t a t i m e . S i m u l t a n e o u s l y t r a n s m i t t i n g m u l t ip l e c o n c e p t s th a t s h a r e d

u n i t s w o u l d c a u s e c r o s s - t a l k . I t i s s t i l l t r u e i n o u r C M t h a t m a n y r e l a t e d

u n i t s w i l l b e t r i g g e r e d b y s p r e a d i n g a c t i v a t i o n , b u t t h e r e p r e s e n t a t i o n o f

e a c h c o n c e p t i s t a k e n t o b e c o m p a c t .

M o s t c o g n i t i v e s c ie n t is ts b e l i e v e t h a t t h e b r a i n a p p e a r s t o b e m a s s i v e l y

p a r a l le l a n d t h a t s u c h s t r u c t u r e s c a n c o m p u t e s p e c ia l f u n c t i o n s v e r y w e l l.

B u t m a s s i v e l y p a r a l l e l s t r u c t u r e s d o n o t s e e m t o b e u s a b l e f o r g e n e r a l p u r -

p o s e c o m p u t i n g a n d t h e r e is n o t n e a r l y a s m u c h k n o w l e d g e o f h o w t o c o n -

s t ru c t a n d a n a l y z e s u c h m o d e l s . T h e c o m m o n b e l i e f w h i c h m a y w e ll b e

r ig h t ) is t h a t t h e r e a r e o n e o r m o r e i n t e r m e d i a t e l e ve ls o f c o m p u t a t i o n a l

o r g a n i z a t i o n l a y e r e d o n th e n e u r o n a l s t r u c t u r e , a n d t h a t t h e o r i e s o f i n te l li -

g e n t b e h a v i o r s h o u l d b e d e s c r i b e d i n t e r m s o f t h e s e h ig h e r - l e v e l l a n g u a g e s ,

s u ch a s P r o d u c t i o n S y s t e m s , P r e d i c a t e C a l c u l u s, o r L I S P . W e h a v e n o t s e e n

a r e d u c t i o n i n t e r p re t e r , if y o u w i ll) o f a n y h i g h e r f o r m a l i s m w h i c h h a s

p l a u s i b l e r e s o u r c e r e q u i r e m e n t s , a n d t h is is a p r o b l e m w e l l w o r t h p u r s u i n g .

O u r a t t e m p t s t o d e v e l o p c o g n i t iv e s c ie n c e m o d e l s d i r e c tl y in n e u r a l

t e r m s m i g h t f a il f o r o n e o f t w o r e a s o n s . I t m a y b e t h a t t h e r e r e a l l y is a n i n -

t e r p r e t e d s y m b o l s y st e m i n a n im a l b r a i n s . I n t hi s ca s e w e w o u l d h o p e t h a t

o u r e f f o r t s w o u l d b r e a k d o w n i n a w a y t h a t c o u l d s h e d l i g h t o n t h e n a t u r e

o f t h is s y m b o l s y s t em . T h e o t h e r p o s s ib i li ty is t h a t C M t e c h n i q u e s a r e

d i r e ct l y a p p l ic a b l e b u t w e a r e u n a b l e t o f ig u r e o u t h o w t o m o d e l s o m e i m -

p o r t a n t c a p a c i t y , e .g . , p l an n i n g . O u r p r o g r a m is t o c o n t i n u e t h e C M a t t a c k

o n p r o b l e m s o f in c r ea s in g d i f f i c u l ty a n d t o i n d u c e s o m e o f y o u t o j o i n u s )

u n t il w e e n c o u n t e r o n e t h a t is i n t r a c t a b l e in o u r t e r m s . T h e r e a r e a n u m b e r

o f p r o b l e m s t h a t a r e k n o w n t o b e d i f f i c u lt f o r sy s t em s w i t h o u t a n i n t e r p r e t e d

s y m b o l i c r e p r e s e n t a t i o n , i n c l ud i n g c o m p l e x c o n c e p t s , le a r n i n g , a n d n a t u r a l

l a n g u a g e u n d e r s t a n d i n g . T h e c u r r e n t p a p e r is m a i n l y c o n c e r n e d w i t h la y in g

o u t t h e f o r m a l i s m a n d s h o w i n g h o w i t a p p l i e s i n t h e e a s y c a s es , b u t w e d o

a d d r e s s t he p r o b l e m o f c o m p l e x c o n c e p t s in S e c t io n 4 . W e h a v e m a d e s o m e

p r o g r e s s o n t h e p r o b l e m o f l e a rn i n g i n C M s y s te m s F e l d m a n , 1 98 1b ) a n d

a r e b e g i n n i n g t o w o r k s e r io u s l y o n n a t u r a l l a n g u a g e p r o c e s si n g a n d o n

h i g h er -l e v el v i si o n . O u r e f f o r t s o n p l a n n i n g a n d l o n g - t e r m m e m o r y r e o r g a -

n i z a t io n h a v e n o t a d v a n c e d s i g n if i c an t l y b e y o n d t h e d i s cu r s iv e p r e s e n t a t i o n

i n F e l d m a n , 1 9 8 0) .

W e w i l l c e r t a i n l y n o t g e t v e r y f a r i n t h i s p r o g r a m w i t h o u t d e v e l o p i n g

s o m e s y s t e m a t ic m e t h o d s o f a tt a c k i n g C M t a sk s a n d s o m e b u i ld i n g - b l o c k

c i r c u i t s w h o s e p r o p e r t i e s w e u n d e r s t a n d . A f i r s t s t e p t o w a r d s a s y s t e m a t i c

d e v e l o p m e n t o f C M is t o d e f i n e a n a b s t r a c t c o m p u t i n g u n i t . O u r u n i t is


7/50

C O N N E C T I O N I S T M O D E L S A N D T H EIR P RO P ER TIE S 2

r a t h e r m o r e g e n e r a l t h a n p r e v i o u s p r o p o s a l s a n d is i n t e n d e d t o c a p t u r e t h e

c u r r e n t u n d e r s t a n d i n g o f t h e i n f o r m a t i o n p r o c e s s i n g c a p a b i l i t i e s o f

n e u r o n s . S o m e u s e f u l s p e ci a l c a se s o f o u r g e n e r a l d e f i n i t io n a n d s o m e p r o p -

e r ti es o f v e r y s i m p l e n e t w o r k s a r e d e v e l o p e d i n S e c t i o n 2 . A m o n g t h e k e y

i d e a s a r e l o c a l m e m o r y , n o n - h o m o g e n e o u s a n d n o n - l i n e a r f u n c t i o n s , a n d

t h e n o t i o n s o f m u t u a l i n h i b i ti o n a n d s t ab l e c o a li t io n s .

A m a j o r p u r p o s e o f t h e re s t o f t h e p a p e r is t o d e s c r ib e b u i ld i n g b l o c k s

w h i c h w e h a v e f o u n d u s e f u l i n c o n s t r u c t i n g C M s o l u t i o n s t o v a r i o u s t a s k s .

T h e c o n s t r u c t i o n s a r e i n t e n d e d t o b e u s e d t o m a k e s p e c if ic m o d e l s b u t t h e

e x a m p l e s i n t h is p a p e r a r e o n l y s u g g es ti v e. W e p r e s e n t a n u m b e r o f C M

s o l u t i o n s t o g e n e r a l p r o b l e m s a r i s i n g i n i n t e l l i g e n t b e h a v i o r , b u t

we are not

suggesting that any o f these are necessarily employed by nature. O u r n o t i o n

o f a n a d e q u a t e m o d e l is o n e t h a t a c c o u n t s f o r

all

o f t h e e s t a b l i s h e d r e l e v a n t

f i n d i n g s a n d t h i s is n o t a ta s k t o b e u n d e r t a k e n l ig h t ly . W e a r e d e v e l o p i n g

s o m e p r e l i m i n a r y s k e t c h e s (B a l l a r d & S a b b a h , 1 9 81 ; S a b b a h , 1 9 81 ) f o r a

s e r io u s m o d e l o f l o w a n d i n t e r m e d i a t e l e ve l v i si o n . A s w e d e v e l o p v a r i o u s

b u i ld i n g b l o c k s a n d t e c h n i q u e s w e w ill al so b e t r y i n g t o b u r y s o m e o f th e

c o n t a m i n a t e d d e b ri s o f p as t n e u r a l m o d e l i n g e f f o r ts . M a n y o f o u r c o n s t ru c -

t io n s a re i n te n d e d a s an s w e rs t o k n o w n h a r d p r o b l e m s in C M c o m p u t a t i o n .

A m o n g t h e i s s u e s a d d r e s s e d a r e : s t a b i l i t y a n d n o i s e - s e n s i t i v i t y , d i s t r i b u t e d

d e c i s i o n - m a k i n g , t i m e a n d s e q u e n c e p r o b l e m s , a n d t h e r e p r e s e n t a t i o n o f

c o m p l e x c o n c e p t s . T h e c r u c ia l q u e s ti o n s o f l e a rn i n g a n d c h a n g e i n C M

s y s t e m s a r e d i s c u s s e d e ls e w h e r e ( F e l d m a n , 1 9 8 1b ) .

2 . N E U R O N - L I K E C O M P U T I N G U N I T S

A s p a r t o f o u r e f f o r t to d e v e l o p a g en e r a ll y u s e fu l f r a m e w o r k f o r c o n n e c -

t io n i s t th e o r i e s , w e h a v e d e v e l o p e d a s t a n d a r d m o d e l o f t h e i n d i v id u a l u n i t .

I t w ill t u r n o u t t h a t a u n i t m a y b e u s e d t o m o d e l a n y t h i n g f r o m a s m a l l

p a r t o f a n e u r o n t o t h e e x t e r n a l f u n c t i o n a l i t y o f a m a j o r s u b s y s t e m . B u t t h e

b a si c n o t i o n o f u n i t is m e a n t t o lo o s e l y c o r r e s p o n d t o a n i n f o r m a t i o n p r o -

c es si ng m o d e l o f o u r c u r r e n t u n d e r s t a n d i n g o f n e u r o n s . T h e p a r t i c u la r

d e f i n i t io n s h e r e w e r e c h o s e n t o m a k e it e a s y to s p e c i f y d e t a il e d e x a m p l e s o f

r e la t iv e l y c o m p l e x b e h a v i o r s . T h e r e i s n o a t t e m p t t o b e m i n i m a l o r m a t h e -

m a t i c a l l y e l e g a n t. T h e v a r i o u s n u m e r i c a l v a l ue s a p p e a r i n g i n t h e d e f i n i ti o n s

a r e a r b i t r a r y , b u t f i x ed f in i t e b o u n d s p l a y a c r u c i a l r o l e i n t h e d e v e l o p m e n t .

T h e p r e s e n t a t i o n o f t h e d e f i n i t i o n s w i ll b e in s ta g e s, a c c o m p a n i e d b y e x-

a m p l e s . A c o m p a c t t e c h n i c a l s p e c i f ic a t i o n f o r r e f e r e n c e p u r p o s e s is i n c l u d e d

a s A p p e n d i x A . E a c h u n i t w ill b e c h a r a c t e r i z e d b y a s m a l l n u m b e r o f d is -

c r e t e s t a t e s p l u s :

p - - a c o n t i n u o u s v a l u e i n [ - 10 , 10], ca l le d

potential

( a c c u r a c y o f s e v e r a l d i g it s )

v - - a n output value i n t e g e r s O _ v _ 9

i - - a v e c t o r o f inputs i , . . . . . in


8/50

2 2 FELDMAN AND BALLARD

P U n i ts

F o r s o m e a p p l i c a t i o n s , w e w i l l b e a b l e t o u s e a p a r t i c u l a r l y s i m p l e k i n d o f

u n i t w h o s e o u t p u t v is p r o p o r t i o n a l t o i ts p o t e n t i a l p ( r o u n d e d ) w h e n p > 0

a n d w h i ch h a s o n l y o n e s t at e . I n o t h e r w o r d s

p - - p + /3

Z; w kik

V -- if p> O then r o u n d ( p - 0 ) e /s e 0

[0_~ wk_< 1]

[v = 0 . . .91

w h e r e / ~ , 0 a r e c o n s t a n t s a n d w ~ a r e w e i g h t s o n t h e i n p u t v a l u e s . T h e w e i g h t s

a r e th e s o l e l o c u s o f c h a n g e w i t h e x p e r i e n c e i n t h e c u r r e n t m o d e l . M o s t

o f t e n , t h e p o t e n t i a l a n d o u t p u t o f a u n i t w il l b e e n c o d i n g i ts confidence a n d

w e w ill s o m e t i m e s u s e t hi s t e r m . T h e - - n o t a t i o n is b o r r o w e d f r o m th e

a s si g nm e n t s t a te m e n t o f p r o g r a m m i n g l an g u a ge s . T h i s n o t a t i o n c o v e r s b o t h

c o n t i n u o u s a n d d i s cr e te t i m e f o r m u l a t i o n s a n d a l l o w s u s t o t a lk a b o u t s o m e

i ss ue s w i t h o u t a n y e x p l i c it m e n t i o n o f ti m e . O f c o u r s e , c e r t a i n o t h e r q u e s -

t io n s w ill i n h e r e n t l y in v o l v e t i m e a n d c o m p u t e r s i m u l a t i o n o f a n y n e t w o r k

o f u n i t s w i l l r a i s e d e l i c a t e q u e s t i o n s o f d i s c r e t i z i n g t i m e .

T h e r e s t r i c t i o n t h a t o u t p u t t a k e o n s m a l l i n t e g e r v a l u e s i s c e n t r a l t o

o u r e n t e r p r is e . T h e f i r i n g f r e q u e n c ie s o f n e u r o n s r a n g e f r o m a f e w to a f e w

h u n d r e d i m p u ls e s p e r s e c o n d . I n t h e 1 / 1 0 s e c o n d n e e d e d f o r b a s ic m e n t a l

e v e n t s , t h e r e c a n o n l y b e a l i m i t e d a m o u n t o f i n f o r m a t i o n e n c o d e d i n f r e -

q u e n c ie s . T h e t en o u t p u t v a l ue s ar e a n a t t e m p t t o c a p t u r e t h is i d ea . A m o r e

a c c u r a t e r e n d e r i n g o f n e u r a l e v e n t s w o u l d b e t o a l l o w 1 0 0 d i s c r e t e v a l u e s

w i t h n o i se o n t r a n s m i s s i o n ( c f. S e j n o w s k i , 1 9 77 ). T r a n s m i s s i o n t i m e is

a s s u m e d t o b e n eg l ig i b le ; d e l a y u n i t s c a n b e a d d e d w h e n t r a n s i t t i m e n e e d s

t o b e t a k e n i n t o a c c o u n t .

T h e p - u n i t is s o m e w h a t l ik e c la s si ca l l i n e a r t h r e s h o l d e l e m e n t s ( M i n s k y

& P a p e r t , 1 9 7 2 ) , b u t t h e r e a r e s e v e r a l d i f f e r e n c e s . T h e p o t e n t i a l , p , i s a

c r u d e f o r m o f m e m o r y a n d i s a n a b s t r a c t i o n o f th e in s t a n t a n e o u s m e m b r a n e

p o t e n t i a l t h a t c h a r a c t e r i z e s n e u r o n s ; i t g r e a t l y r e d u c e s t h e n o i s e s e n s it i v it y o f

o u r n e t w o r k s . W i t h o u t l o c a l m e m o r y i n t h e u n i t , o n e m u s t g u a r a n t e e t h a t

a ll t h e i n pu t s r e q u i r e d f o r a c o m p u t a t i o n a p p e a r s i m u l t a n e o u s l y a t th e u n i t .

O n e p r o b l e m w i t h t h e d e f i n i t i o n a b o v e o f a p - u n i t i s t h a t i t s p o t e n t i a l

d o e s n o t d e c a y i n t h e a b s e n c e o f i n p u t . T h i s d e c a y i s b o t h a p h y s i c a l p r o p -

e r t y o f n e u r o n s a n d a n i m p o r t a n t c o m p u t a t i o n a l f e a tu r e f o r o u r h ig h ly

p a r a l l e l m o d e l s . O n e c o m p u t a t i o n a l t r i c k t o s o l v e t h i s i s t o h a v e a n i n -

h i b i t o r y c o n n e c t i o n f r o m t h e u n i t b a c k t o i t se lf . I n f o r m a l l y , w e i d e n t i f y t h e

n e g a t i v e s e lf f e e d b a c k w i t h a n e x p o n e n t i a l d e c a y in p o t e n t i a l w h i c h i s

m a t h e m a t i c a l l y e q u i v a l e n t . W i t h t h is a d d i t i o n , p - u n it s c a n b e u s e d f o r

m a n y C M t as k s o f i n t e r m e d i a t e d i f f ic u l t y . T h e I n t e r a c t i v e A c t i v a t i o n

m o d e l s o f M c C l e l l a n d a n d R u m e l h a r t c a n b e d e s c r i b e d n a t u r a l l y w i t h

p - u n it s , a n d s o m e o f o u r o w n w o r k ( B a l l a rd , 1 98 1) a n d t h a t o f o t h e r s ( M a r r


9/50

C ON N EC T I ON I ST M OD ELS A N D T H EI R PR OPER TIES 21 3

& P o g g i o , 1 97 6) ca n b e d o n e w i th p - u n i t s . B u t th e r e a r e a n u m b e r o f a d d i -

t io n a l f e a t u r e s w h ic h w e h a v e f o u n d v a l u a b l e in m o r e c o m p l e x m o d e l i n g

t a s k s .

D i s j u n c t iv e F i r in g o n d i t io n s a n d o n j u n c t iv e o n n e c t i o n s

I t i s b o t h c o m p u t a t i o n a l l y e f f i c i e n t a n d b i o l o g i c a l l y r e a l i s t i c t o a l l o w a u n i t

t o r e s p o n d t o o n e o f a n u m b e r o f a l te r n a t i v e c o n d i t i o n s . O n e w a y t o vi e w

t hi s is t o i m a g i n e t h e u n it h a v i n g d e n d r i t e s e a c h o f w h i c h d e p ic t s a n a lt e r-

n a t i v e e n a b l in g c o n d i t in n ( F ig u r e 3 ). F o r e x a m p l e , o n e c o u l d e x t e n d t h e n e t-

w o r k o f F i g u r e 1 t o a l lo w f o r s ev e r a l d i f f e r e n t t y p e f o n t s a c t i v a t in g t h e s a m e

l e tt e r n o d e , w i t h t h e h i g h e r c o n n e c t i o n s u n c h a n g e d . B i o l o g i c a l ly , t h e f ir i n g

o f a n e u r o n d e p e n d s , in m a n y c as e s , o n lo c al s p a t i o - t e m p o r a l s u m m a t i o n

i n v o l v in g o n l y a s m a l l p a r t o f t h e n e u r o n ' s s u r f a c e . S o - c a ll e d d e n d r i t ic

s p i k e s t r a n s m i t t h e a c t i v a t i o n t o t h e r e s t o f t h e c e l l .

i

i

is

i 8

i7

F i g u r e 3 C o n j u n c t i v e c o n n e c t i o n s a n d d i s ju n c t i v e i n p u t s i t e s

I n te r m s o f o u r f o r m a l i s m , t h i s c o u l d b e d e s c r ib e d in a v a r i e t y o f

w a y s . O n e o f t h e s i m p l e st is t o d e f i n e th e p o t e n t i a l i n t e r m s o f t h e m a x i m u m

o f t h e s e p a r a te c o m p u t a t i o n s , e . g . ,

p - - p + /~ M ax ( i , + i2 - ~ , i3 + i , - ~ , is + i6 - i7 - )

w h e r e / 3 is a s c al e c o n s t a n t a s in t h e p - u n i t a n d i s a c o n s t a n t c h o s e n ( u s u a l l y

> 10) t o s u p p r e s s n o i s e a n d r e q u i r e t h e p r e s e n c e o f m u l t i p l e a c t i v e i n p u t s

( S a b b a h , 1 9 81 ). T h e m i n u s s i gn a s s o c i a t e d w i th i , c o r r e s p o n d s t o i ts b e i n g

a n i n h i b i t o r y i n p u t .

I t d o e s n o t s e e m u n r e a s o n a b l e ( g iv e n c u r r e n t d a t a , K u f f l e r & N ic h o l ls ,

1 97 6) to m o d e l t h e f i r i n g r a te o f s o m e u n i t s a s th e m a x i m u m o f t h e r a te s a t

its a ct iv e si te s. U n i t s w h o s e p o t e n t i a l is c h a n g e d a c c o r d i n g t o th e m a x i m u m

o f a s e t o f a l g e b r a i c s u m s w il l o c c u r f r e q u e n t l y i n o u r s p e c i fi c m o d e l s . O n e

a d v a n t a g e o f k e e p i n g th e p r o c e s s i n g p o w e r o f o u r a b s t r a c t u n i t c lo s e t o th a t

o f a n e u r o n is t h a t i t h e l ps i n f o r m o u r c o u n t i n g a r g u m e n t s . W h e n w e a t -


10/50

2 4 F E L D M A N A N D B AL LA RD

t e m p t t o m o d e l a p a r t i c u l a r f u n c t i o n ( e . g ., s t e r e o p s i s) , w e e x p e c t t o r e q u i r e

t h a t t h e n u m b e r o f u n i t s a n d c o n n e c t i o n s a s w e ll a s t h e e x e c u t i o n t im e r e-

q u i r e d b y t h e m o d e l a r e p l a u s i b l e .

T h e m a x - o f - s u m u n it is t h e c o n t i n u o u s a n a l o g o f a lo g ic a l O R - o f - A N D

( d i sj u n c ti v e n o r m a l f o r m ) u n i t a n d w e w ill s o m e t i m e s u s e t h e l a t t e r as a n a p -

p r o x i m a t e v e r si o n o f t h e f o r m e r . T h e O R - o f - A N D u n i t c o r re s p o n d i n g t o

F ig u re 3 i s :

p - p + ot O R ( i , i2 , i3 i , , i s i~ (no t i , ) )

T h i s f o r m u l a t i o n s tr es se s t h e i m p o r t a n c e t h a t n e a r b y s p a t i a l c o n n e c t i o n s ll

b e f i r i n g b e f o r e t h e p o t e n t i a l i s a f f e c t e d . H e n c e , i n t h e a b o v e e x a m p l e , i 3

a n d i4 m a k e a

conjunctive connection

w i t h t h e u n it . T h e e f f e c t o f a c o n j u n c -

t iv e c o n n e c t i o n c a n a l w a y s b e s im u l a t e d w i th m o r e u n i t s b u t t h e n u m b e r o f

e x t r a u n i t s m a y b e v e r y l a r g e .

Q-Uni ts and Compound Uni ts

A n o t h e r u s e f u l sp e c ia l c as e ar is e s w h e n o n e s u p p r e s s e s t h e n u m e r i c a l p o t e n -

t ia l , p , a n d r e li es u p o n a f i n i t e - s t a t e s e t { q } f o r m o d e l i n g . I f w e a l s o i d e n t i f y

e a c h i n p u t o f i w i t h a s e p a r a t e n a m e d i n p u t s i g n a l , w e c a n g e t cl a ss i ca l f i n it e

a u t o m a t a . A s i m p le e x a m p l e w o u l d b e a u n i t t h a t c o u l d b e s t a r t e d o r s t o p p e d

f r o m f i r i n g .

O n e c o u l d d e s c r ib e t h e b e h a v i o r o f t h i s u n i t b y a t a b l e , w i th r o w s c o r -

r e s p o n d i n g t o s t a t e s i n { q } a n d c o l u m n s t o p o s s i b l e i n p u t s , e . g . ,

i~ (s ta rt ) i2 s top)

F i r i n g F i r i n g N u l l

N u l l N u l li r i n g

O n e w o u l d a l s o h a v e t o s p e c if y a n o u t p u t f u n c t i o n , g i v i n g o u t p u t v a l u e s r e -

q u i r e d b y t h e r e s t o f t h e n e t w o r k , e . g . ,

v - - i f q = F i r i n g t h e n 6 e ls e 0.

T h i s c o u l d a l s o b e a d d e d t o t h e t a b l e a b o v e . A n e q u i v a l e n t n o t a t i o n w o u l d

b e t r a n s i ti o n n e t w o r k s w i t h s ta t es a s n o d e s a n d i n p u t s a n d o u t p u t s o n t h e

a r c s .

I n o r d e r t o b u i l d m o d e l s o f i n t e r e s t i n g b e h a v i o r s w e w i l l n e e d t o

e m p l o y m a n y o f t h e s a m e t e c h n iq u e s u s e d b y d e si gn e rs o f c o m p l e x c o m -

p u t e r s a n d p r o g r a m s . O n e o f th e m o s t p o w e r f u l t e c h n i q u e s w i ll b e e n c a p s u -


11/50

CONNECTIONISTMODELSAN D THEIRPROPERTIES 215

lation and abst raction o f a subnetwork by an individual unit. For example,

a system that had separate motor abilities for turning eft and turning right

(e.g., fins) could use two start-stop units to model a turn-unit, as shown in

Figure 4.

l e f t start ~ c u s e s

s t o p ~ ~ ~ m ot ion

t o left

~ ~ s t e c u s e s

righ t rt . ~ -~

m o t i o n

~ . s t o p ~ t o r ig h t

Figure 4. A Turn Unit .

Note that the compound unit here has two distinct outputs, where

basic units have only one (which can branch, of course). In general, com-

pound units will differ from basic ones only in that they can have several

distinct outputs.

The main point of this example is that the turn-unit can be described

abstractly, independent of the details of how it is built. For example, using

the tabular conventions described above,

e f t R i g h t V a l u e s O u t p u t

a gauche a gauche ad ro i t v, =7 , v2=O

ad ro it a gauche ad ro it v, =0, v2=8

where the right-going output being larger than the left could mean that we

have a right-finned robot . There is a great deal more that must be said about

the use of states and symbolic input names, about multiple simultaneous in-

puts, etc., but the idea of describing the external behavior of a system only

in enough detail for the task at hand is of great importance. This is one of

the few ways known of coping with the complexity of the magnitude needed

for serious modeling of biological functions . It is not strictly necessary that

the same formalism be used at each level of functional abstraction and, in

the long run, we may need to employ a wide range of models. For example,

for certain purposes one might like to expand our units in terms of compar t-

mental models of neurons like those of (Perkel, 1979). The advantage of

keeping within the same formalism is that we preserve intuition, mathe-

matics, and the ability to use existing simulation programs. With sufficient

care, we can use the units defined above to represent large subsystems with-


12/50

2 6 F E L D M A N A N D B AL LA RD

o u t g iv i n g u p t h e n o t i o n t h a t e a c h u n i t c a n s t a n d f o r a n a b s t r a c t n e u r o n . T h e

c r u c i a l p o i n t i s t h a t a s u b s y s t e m m u s t b e e l a b o r a t e d i n t o it s n e u r o n - l e v e l

u n i ts f o r t i m i n g a n d s i ze c a l c u l a ti o n s , b u t c a n h o p e f u l l y ) b e d e s c r i b e d m u c h

m o r e s i m p l y w h e n o n l y i ts e f f e c ts o n o t h e r s u b s y s t e m s a r e o f d i re c t c o n c e r n .

U n i t s E m p l o y i n g p a n d q

I t w i ll a l r e a d y h a v e o c c u r r e d t o t h e r e a d e r t h a t a n u m e r i c a l v a l u e , l ik e o u r p ,

w o u l d b e u s e f u l f o r m o d e l i n g t h e a m o u n t o f t u r n i n g t o th e le f t o r r ig h t i n

t h e l a s t e x a m p l e . I t a p p e a r s t o b e g e n e r a l l y t r u e t h a t a s in g l e n u m e r i c a l v a l u e

a n d a s m a l l se t o f d i s c r e te s t at e s c o m b i n e t o p r o v i d e a p o w e r f u l y e t t r a c t a b l e

m o d e l i n g u n i t . T h i s is o n e r e a s o n t h a t t h e c u r r e n t d e f i n i ti o n s w e r e c h o s e n .

A n o t h e r r e a s o n is t h a t t h e m i x e d u n i t s e e m s t o b e a p a r t i c u l a r l y c o n v e n i e n t

w a y o f m o d e l in g t h e i n f o r m a t i o n p r o c es s in g b e h a v i o r o f n e u r o n s , a s g e n e r -

a l ly d e s c r ib e d . T h e d i s c r e te s t a te s e n a b l e o n e t o m o d e l t h e e f f e c t s i n n e u r o n s

o f p o l y p e p t id e m o d u l a t o r s , a b n o r m a l c h e m i c a l e n v i r o n m e n t s , f a ti g u e, e tc .

A l t h o u g h t h e s e e f f e c ts a r e o f t e n c o n t i n u o u s f u n c t i o n s o f un i t p a r a m e t e r s ,

t h e r e a r e s e v e r a l a d v a n t a g e s t o u s i n g d i s c r e t e s t a t e s in o u r m o d e l s . S c i e n ti s ts

a n d l a y m e n al ik e o f t e n g i v e d i st in c t n a m e s e . g . , c o o l , w a r m , h o t ) t o p a r a m -

e t e r r a n g e s t h a t t h e y w a n t t o t r e a t d i f f e r e n t l y . W e a l s o c a n e x p l o i t a l a r g e

l i te r a t u re o n u n d e r s t a n d i n g l o o s e l y - c o u p l e d s y s t e m s a s f i n it e - st a te m a c h i n e s

S u n s h i n e , 1 97 9). I t is a l s o t r a d i t i o n a l t o b r e a k u p a f u n c t i o n i n t o s e p a r a t e

r a n g e s w h e n i t is s i m p l e r t o d e s c r ib e t h a t w a y . W e h a v e a l r e a d y e m p l o y e d a l l

o f th e s e u se s o f d is c r et e s t a te s i n o u r d e t a il e d w o r k F e l d m a n , 1 9 81 b ; S a b b a h ,

1 98 1). O n e e x a m p l e o f a u n it e m p l o y i n g b o t h p a n d q n o n - t r i v i a l l y is t h e

f o l lo w i n g c r u d e n e u r o n m o d e l . T h i s m o d e l is c o n c e r n e d w i th s a t u r a t i o n a n d

a s s u m e s t h a t t h e o u t p u t s t r e n g t h , v , is s o m e t h i n g l ik e a v e r a g e f i r i n g f re -

q u e n c y . I t is n o t a m o d e l o f in d i v i d u a l a c t io n p o t e n t i a l s a n d r e f r a c t o r y

p e r i o d s .

W e s u p p o s e t h e d is t in c t s t a t e s o f t h e u n i t q e { n o r m a l , r e c o v e r } . I n

norm l

s t a t e t h e u n i t b e h a v e s l i k e a p - u n i t , b u t w h i l e i t i s

recovering

i t ig-

n o r e s i n p u t s . T h e f o l lo w i n g t a b l e c a p t u r e s a l m o s t a l l o f t hi s b e h a v i o r .

i n c o m p l e t e )

n o r m a l

r e c o v e r

1 < p < 9 p > 9 O u t p u t V a l u e

p - - p + E i p - - - p / v - -c t p - / 9

r e c o v e r

n o r m a l < i m p o s s i b le > v - - O

H e r e w e h a v e t he c h a n g e f r o m o n e s t a te t o th e o t h e r d e p e n d i n g o n t h e

v a l u e o f t h e p o t e n t i a l , p , r a t h e r t h a n o n s p e c i f ic i n p u t s . T h e r e c o v e r i n g s t a t e

is a l s o c h a r a c t e r i z e d b y t h e p o t e n t i a l b e i n g s e t n e g a t i v e . T h e u n s p e c i f i e d

is su e is w h a t d e t e r m i n e s t h e d u r a t i o n o f th e r e c o v e r i n g s t a t e - - t h e r e a r e


13/50

CONNECTIONIST MODELS A N D THEIR PROPERTIES 2 7

s e v e r a l p o s s i b i l it i e s . O n e is a n e x p l i c i t d i s h a b i t u a t i o n s i g n a l l i k e t h o s e i n

K a n d e l s e x p e r i m e n t s ( K a n d e l , 1 97 6) . A n o t h e r w o u l d b e t o h a v e th e u n i t

s u m i n p u t s i n t h e r e c o v e r i n g s ta t e a s w e ll . T h e r e a d e r m i g h t w a n t t o c o n -

s i d e r h o w t o a d d t h i s t o t h e t a b l e .

T h e t h i r d p o s s i b il i ty , w h i c h w e w i ll u s e f r e q u e n t l y , is t o a s s u m e t h a t

t h e p o t e n t i a l , p , d e c a y s t o w a r d z e r o ( f r o m b o t h d i r e c t i o n ~ ) u n l e s s e x p l i c i t l y

c h a n g e d . T h i s i m p l i c it d e c a y p - - p 0 e -kf c a n b e m o d e l e d b y s e l f i n h i b i t i o n ; t h e

d e c a y c o n s t a n t , k , d e t e r m i n e s t h e l e n g th o f t h e r e c o v e r y p e r i o d .

T h e g e n e r al d e f i n i t i o n o f o u r a b s t r a c t n e u r a l c o m p u t i n g u n i t is j u s t a

f o r m a l i z a t i o n o f th e i d e as p r e s e n t e d a b o v e . T o th e p r e v i o u s n o t i o n s o f p , v ,

a n d i w e f o r m a l l y a d d

{ q } - - a s e t o f

discrete s tates

< 10

a n d f u n c t i o n s f r o m o l d t o n e w v a l u e s o f t h es e

p - - f ( i , p , q )

q - - g ( i , p , q )

v - - h ( i , p , q )

w h i ch w e a s su m e , f o r n o w , t o c o m p u t e c o n t i n u o u s l y . T h e f o r m o f t h e f , g ,

a n d h f u n c t i o n s w il l v a r y , b u t w i ll g e n e r a l l y b e r e s t r i c t e d t o c o n d i t i o n a l s a n d

s i m p l e f u n c t i o n s . T h e r e a r e b o t h b i o l o g i c a l a n d c o m p u t a t i o n a l r e a s o n s f o r

a l l o w i n g u n i ts t o r e s p o n d ( f o r e x a m p l e ) l o g a r i t h m i c a l l y t o t h e i r i n p u t s a n d

w e h a v e a l re a d y s ee n i m p o r t a n t u s e s o f th e m a x i m u m f u n c t i o n .

T h e o n l y o t h e r n o t i o n t h a t w e w ill n e e d is m o d i f i e r s a s s o c ia t e d w i t h

t h e i n p u t s o f a u n i t . W e e l a b o r a t e t h e i n p u t v e c t o r i i n t e r m s o f r e c e i v e d

v a l u e s , w e i g h t s , a n d m o d i f i e r s :

V j , b = r~-w~..mj j = 1 . . . . n

w h e r e r j i s t h e value r ec e iv e d f r o m a p re d e c e s so r [ r = 0 . . . 9 ] ; w j is a

c h a n g e a b l e

weight

u n s ig n e d [0 _< w~_< 1] ( a c c u r a c y o f s e v e ra l d ig i t s ) ; a n d m j

i s a s y n a p t o - s y n a p t i c m o d i f i e r wh ic h i s e i t h e r 0 o r 1 .

T h e w e i gh t s a r e th e o n l y t h i n g i n th e s y s t e m w h i c h c a n c h a n g e w i th e x -

p e r i en c e . T h e y a r e u n s i g n e d b e c a u s e w e d o n o t w a n t a c o n n e c t i o n to c h a n g e

f r o m e x c i t a t o r y to i n h i b i t o r y . T h e m o d i f i e r o r g a te s im p l i fi e s m a n y o f o u r

d e t a i l e d m o d e l s . L e a r n i n g a n d c h a n g e w i ll n o t b e t r e a t e d t e c h n i c a l l y i n t h is

p a p e r , b u t t h e d e f i n i t i o n s a r e i n c l u d e d i n t h e A p p e n d i x f o r c o m p l e t e n e s s

( F e l d m a n , 1 9 8 1 b ) .

W e c o n c l u d e t h is s e c t io n w i t h s o m e p r e l i m i n a r y e x a m p l e s o f n e t w o r k s

o f o u r u n i t s , i l l u s tr a t i n g t h e k e y id e a o f m u t u a l ( l a te r a l ) i n h i b i t i o n ( F i g . 5 ).

M u t u a l i n h i b i ti o n i s w i d e s p r e a d i n n a t u r e a n d h a s b e e n o n e o f t h e b as ic

c o m p u t a t i o n a l s c h e m e s u s e d i n m o d e l i n g . W e w i l l p r e s e n t t w o e x a m p l e s o f


14/50

2 8 FELDMAN AND BALLARD

h o w i t w o r k s t o h e l p a id i n i n t u i t i o n a s w e l l a s t o i l lu s t r a t e t h e n o t a t i o n . T h e

b a s ic s i t u a t io n is sy m m e t r i c c o n f i g u r a t i o n s o f p - u n i t s w h i c h m u t u a l l y i n-

h i b i t o n e a n o t h e r . T i m e is b r o k e n i n t o d i s c r e t e i n t e r v a ls f o r th e s e e x a m p l e s .

T h e e x a m p l e s a r e t o o s i m p l e t o b e r e a l i s t i c , b u t d o c o n t a i n i d e a s w h i c h w e

w i l l e m p l o y r e p e a t e d l y .

T w o P U n i t s S y m m e t r i c a l l y C o n n e c t e d

S u pp o se w, = 1 , w2 = - .5

p( t + 1) = p( t ) + r , - ( .5)r2

v = r o u n d ( p ) [ 0 . . . 9 ]

rs = r e c e iv e d

R e f e r r i n g t o F i g u r e 5 a , s u p p o s e t h e i n i ti a l i n p u t t o t h e u n i t A . 1 is 6 , t h e n 2

p e r t i m e s t e p , a n d t h e i n it ia l i n p u t t o B . I is 5 , t h e n 2 p e r t i m e s t e p . A t e a c h

t i m e s t e p , e a c h u n i t c h a n g e s i t s p o t e n t i a l b y a d d i n g t h e e x t e r n a l v a l u e ( r , )

a n d s u b s t r a c t i n g h a l f t h e o u t p u t v a l u e o f i ts r iv a l . T h i s s y s t e m w i ll s t a b il i ze

t o t h e s id e o f t h e l a rg e r o f t w o i n s t a n t a n e o u s i n p u t s.

T w o S y m m e t r i c C o a l i t i o n s o f 2 U n i t s

w l = l

W ~ = . 5

W3 = .5

p( t + 1) = p( t ) + r , + .5( r , - r3)

v = r o u n d ( p )

A , C s t a r t a t 6 , B , D a t 5 ;

A , B , C , D h a v e n o e x t e r n a l i n p u t f o r t > 1

T h e c o n n e c t i o n s f o r t hi s s y st e m a r e s h o w n in F i g u r e 5 b . T h i s s y s te m

c o n v e r g e s f a st e r t h a n t h e p r e v i o u s e x a m p l e . T h e i d e a h e r e is t h a t u n i ts A a n d

C f o r m a c o a l i t i o n w i th m u t u a l ly r e i n fo r c i n g c o n n e c t i o n s . T h e c o m p e t i n g

u n i t s a r e A v s . B a n d C v s . D . T h e l a s t e x a m p l e is t h e s m a l l e s t n e t w o r k d e -

p i ct in g w h a t w e b e l ie v e t o b e t h e b a s ic m o d e o f o p e r a t i o n i n c o n n e c t i o n i s t

s y s t e m s . T h e f a s t e r c o n v e r g e n c e is n o t a n a r t i f a c t ; t h e

positive feedb ck

a m o n g m e m b e r s o f a c o a l i t io n w i ll g e n e r a l l y l e ad t o f a s t e r c o n v e r g e n c e t h a n

in s e p a r a t e c o m p e t i t i o n s . I t is t h e a m o u n t o f p o s i ti v e f e e d b a c k r a t h e r t h a n

j u s t t h e s iz e o f t h e c o a l i t io n t h a t d e t e r m i n e s t h e r a t e o f c o n v e r g e n c e ( F el d -

m a n & B a l l a r d , 1 98 2). I n t e r m s o f F i g u r e 1, t h is c o u l d r e p r e s e n t t h e b e h a v i o r

o f t h e ri va l l et te r s A a n d T in c o n j u n c t i o n w i t h t h e ri v al w o r d s A B L E a n d

T R A P , in th e a b s e n c e o f o t h e r a c t i v e n o d e s .

C o m p e t i n g c o a l i ti o n s o f u n i t s w i ll b e th e o r g a n i z i n g p r i n c ip l e b e h i n d

m o s t o f o u r m o d e l s . C o n s i d e r t h e tw o a l t e r n a ti v e r e ad i n g s o f th e N e c k e r


15/50

. Q

Q Q .

u u

0 0

o .

~ ~ o

0

z

u

E

E

E

0

C

o

M.

2 9


16/50

2 2 F E LD MA N A N D B A LL A RD

c u b e s h o w n i n F i g u r e 6. A t e a c h l ev e l o f v i s u a l p r o c e s s i n g , t h e r e a r e m u t u a l l y

c o n t r a d i c t o r y u n it s r e p r e s e n t i n g a l t e r n a t i v e p o s s ib i li ti e s. T h e d a s h e d l in e s

d e n o t e th e b o u n d a r i e s o f c o a li t io n s w h i c h e m b o d y t h e a l t e rn a t i v e in t e r p r e -

t a t io n s o f t h e im a g e . A n u m b e r o f i n te r e s ti n g p h e n o m e n a ( e .g . , p r i m i n g ,

p e r c e p t u a l r i v a l r y , f i ll in g , s u b j e c t i v e c o n t o u r ) f i n d n a t u r a l e x p r e s s i o n in th i s

f o r m a l i s m . W e a re e n g a g e d i n an o n g o i n g e f f o r t (B a l l a rd , 1 98 1; S a b b a h ,

1 9 8 1 ) t o m o d e l a s m u c h o f v i s u a l p r o c e s s i n g a s p o s s i b l e w i t h i n t h e c o n n e c -

t i o n is t f r a m e w o r k . T h e n e x t s e c t io n d e s c r ib e s in s o m e d e t a i l a v a r i e t y o f

s i m p l e n e t w o r k s w h i c h w e h a v e f o u n d t o b e u s e f u l i n t h i s e f f o r t .

3 . N E T W O R K S O F U N I T S

T h e m a i n r e s tr i c ti o n i m p o s e d b y t h e c o n n e c t i o n i s t p a r a d i g m is t h a t n o s y m -

b o l i c i n f o r m a t i o n is p a s s e d f r o m u n i t to u n i t . T h i s r e s t r ic t i o n m a k e s i t d i f f i -

c u lt t o e m p l o y s t a n d a r d c o m p u t a t i o n a l d e v i c es l ik e p a r a m e t e r i z e d f u n c t i o n s .

I n t h is s e c t io n , w e p r e s e n t c o n n e c t i o n i s t s o l u t i o n s t o a v a r i e t y o f c o m p u t a -

t i o n a l p r o b l e m s . T h e s e c t i o n s a d d r e s s t w o p r i n c i p a l i s s u e s . O n e i s : C a n t h e

n e t w o r k s b e c o n n e c t e d u p i n a w a y t h a t i s s u f f i c i e n t t o r e p r e s e n t t h e p r o b -

l e m a t h a n d ? T h e o t h e r i s : G i v e n t h e s e c o n n e c t i o n s , h o w c a n t h e n e t w o r k s

e x h i b i t a p p r o p r i a t e d y n a m i c b e h a v i o r , s u c h a s m a k i n g a d e c i s i o n a t a n

a p p r o p r i a t e t i m e ?

U s i n g a U n i t t o e p r e s e n t a V a l u e

O n e k e y to m a n y o f o u r c o n s t r u c t i o n s i s t h e d e d i c a t i o n o f a s e p a r a t e u n i t t o

e a c h v a lu e o f e a c h p a r a m e t e r o f i n te r e st , w h i c h w e te r m t h e u n i t / v a l u e p r i n -

c ip le . W e w ill s h o w h o w t o c o m p u t e u s in g u n i t / v a l u e n e t w o r k s a n d p r e s e n t

a r g u m e n t s t h a t t h e n u m b e r o f u n i ts r e q u i r e d is n o t u n r e a s o n a b l e . I n t h is

r e p r e s e n ta t i o n t h e o u t p u t o f a u n i t m a y b e t h o u g h t o f a s a c o n f i d e n c e m e a -

s u re . S u p p o s e a n e t w o r k o f d e p t h u n i t s e n c o d e s th e d i s t a n c e o f s o m e o b j e c t

f r o m t h e r e t i n a . T h e n i f t h e u n i t r e p r e s e n t i n g d e p t h = 2 s a t u r a t e s , t h e n e t -

w o r k is e x p r e s s i n g c o n f i d e n c e th a t t h e d i s t a n c e is t w o u n i t s . S i m i l a r ly , t h e

G - h i d d e n n o d e in F i g u re 6 e x p r es s e s c o n f i d e n c e in i ts a s s e r ti o n . T h e r e is

m u c h n e u r o p h y s i o l o g i c a l e v i d e n c e t o su g g e st u n i t / v a l u e o r g a n i z a t i o n s i n

l es s a b s t r a c t c o r t i c a l m a p s . E x a m p l e s a r e e d g e s e n s i ti v e u n i t s ( H u b e l &

W i e s e l , 1 9 7 9 ) a n d p e r c e p t u a l c o l o r u n i t s ( Z e k i , 1 9 8 0 ) , w h i c h a r e r e l a t i v e l y

i n se n s it iv e t o i l l u m i n a t i o n s p e c t r a . E x p e r i m e n t s w i th c o r t i c a l m o t o r c o n t r o l

i n t h e m o n k e y a n d c a t ( W u r t z & A l b a n o , 1 98 0) s u g ge s t a u n i t / v a l u e o r g a n i -

z a t i o n . O u r h y p o t h e s i s is t h a t t h e u n i t / v a l u e o r g a n i z a t i o n i s w i d e s p r e a d ,

a n d is a f u n d a m e n t a l d e s i g n p ri n c i p l e .


17/50

CON NEC TIONIST MO DELS A N D THEIR PROPERTIES 221

H

C

\ \

\

\

J A

\

I /

F i gu r e 6 The Ne cke r Cube

A l t h o u g h m a n y p h y si ca l n e u r o n s d o s e e m t o f o l lo w th e u n i t / v a l u e

r u le a n d r e s p o n d a c c o r d i n g t o t h e r e l i ab i li ty o f a p a r t i c u l a r c o n f i g u r a t i o n ,

t h e r e a r e al so o t h e r n e u r o n s w h o s e o u t p u t r e p r e se n t s th e r a n g e o f s o m e

p a r a m e t e r , a n d a p p a r e n t l y s o m e u n i ts w h o s e f i ri n g f r e q u e n c y re f le c t s b o t h

r a n g e a n d s t re n g t h i n f o r m a t i o n S c i e nt if ic A m e r i c a n , 1 97 9). B o t h o f t h e

l a tt e r t y p e s c a n b e a c c o m m o d a t e d w i t h i n o u r d e f i n i t io n o f a u n i t , b u t w e

w i ll e m p l o y o n l y u n i t / v a l u e n e t w o r k s i n t h e r e m a i n d e r o f t h is p a p e r .

I n t h e u n i t / v a l u e r e p r e s e n ta t i o n , m u c h c o m p u t a t i o n is d o n e b y t a bl e

l o o k - u p . A s a s i m p l e e x a m p l e , l e t u s c o n s i d e r t h e m u l t i p l i c a t io n o f t w o v a r i-

a b l e s , i . e . , z = x y . I n t h e u n i t / v a l u e f o r m a l i s m t h e r e w i l l b e u n i t s f o r

e v e r y

v a l u e o f x a n d y t h a t i s i m p o r t a n t . A p p r o p r i a t e p a i rs o f t h e se w ill m a k e a

c o n j u n c t i v e c o n n e c t i o n w i t h a n o t h e r u n i t c e ll r e p r e s e n t in g a s p e c if i c v a l u e

f o r t h e p r o d u c t . F i g u r e 7 s h o w s t h i s f o r a s m a l l se t o f u n i t s r e p r e s e n t i n g

v a l u es f o r x a n d y . N o t i c e th a t t h e c o n f i d e n c e e x p r e ss e d a s o u t p u t v a l u e )

t h a t a p a r t i c u l a r p r o d u c t i s a n a n s w e r c a n b e a l i n e a r f u n c t i o n o f t h e m a x -


18/50

2 2 2 F E L D M A N A N D B A LL AR D

i m u m o f t h e s um s o f t h e c o n f i d e n c es o f i ts t w o i n p ut s . A m a j o r p r o b l e m

w i t h f u n c t i o n ta b l e s a n d w i t h C M i n g e n e r a l) is t h e p o t e n t i a l c o m b i n a t o r i a l

e x p l o s i o n i n t h e n u m b e r o f u n i ts r e q u i r e d f o r a c o m p u t a t i o n . A n a i v e a p -

p r o a c h w o u l d d e m a n d N 2 u n i t s to r e p re s e n t a ll p r o d u c t s o f n u m b e r s f r o m 1

t o N . T h e n e t w o r k o f F i g u r e 7 r e q u i r e s m a n y f e w e r u n i ts b e c a u s e e a c h p r o d -

u c t is r e p r e s e n t e d o n l y o n c e , a n o t h e r a d v a n t a g e o f c o n j u n c t i v e c o n n e c t i o n s .

W e c o u l d u s e e v e n f e w e r u n it s b y e x p l o i ti n g p o s i t i o n a l n o t a t i o n a n d r e p l ac -

i ng e ac h o u t p u t c o n n e c t i o n w i th a c o n j u n c t i o n o f o u t p u t s f r o m u n it s r e p re -

s e n t i n g m u l t i p l e s o f l , 10 , 1 00 , e t c . T h e q u e s t i o n o f e f f i c i e n t w a y s o f

b u i l d i n g c o n n e c t i o n n e t w o r k s i s t r e a t e d i n d e t a i l i n S e c t i o n 4 c f . a l s o H i n -

to n , 1 9 8 1 a ; 1 9 8 1 b ) .

z - - - - f x , y ) = x y

x u n i t s

.onit

z u n i t :

F i g u r e 7. M u l t i p l i c a t i o n U n i ts

odifiers and appings

T h e i d ea o f f u n c t i o n t ab l e s F ig . 7) c a n b e e x t e n d e d t h r o u g h t h e u s e o f vari

able mappings. I n o u r d e f i n i t io n o f t h e c o m p u t a t i o n a l u n i t , w e i n c l u d e d a

b i n a r y m o d i f i e r , m , a s a n o p t i o n o n e v e r y c o n n e c t i o n . A s t h e d e f i n i t i o n

s p e c if ie s , i f t h e m o d i f i e r a s s o c i a t e d w i t h a c o n n e c t i o n is z e r o , t h e v a l u e v

s e n t a lo n g t h a t c o n n e c t i o n is i g n o r e d . T h u s t h e m o d i f i e r d e n o t e s i n h i b i t i o n ,

o r b l o c k i n g . T h e r e i s c o n s i d e r a b l e e v i d e n c e i n n a t u r e f o r s y n a p s e s o n s y n a p -

se s K a n d e l , 1 97 6) a n d t h e m o d i f i e r s a d d g r e a t ly t o th e c o m p u t a t i o n a l

s im p l i c it y o f o u r n e t w o r k s . L e t u s st a r t w i t h a n i n it ia l i n f o r m a l e x a m p l e o f

t h e u se o f m o d i f i e r s a n d m a p p i n g s . S u p p o s e t h a t o n e h a s a m o d e l o f g ra s s as

g r e e n e x c e p t i n C a l i f o r n i a w h e r e i t is b r o w n g o l d e n ) , a s s h o w n i n F i g u r e 8 .


19/50

C ON N EC T I ON I ST M OD E LS A N D T H EIR PR OPER TIES 22 3

F i g u r e 8 G r a s s is G r e e n c o n n e c t i o n m o d i f i e d b y C a l i fo r n i a

H e r e w e c a n s e e t h a t g r a s s a n d g r e e n a re p o t e n t i a l m e m b e r s o f a c o a l i t i o n

c a n r e i n f o r c e o n e a n o t h e r ) e x c e p t w h e n t h e l i n k is b l o c k e d . T h i s u s e is s im i -

l a r t o t h e c a n c e l l a t i o n li n k o f F a h l m a n , 1 97 9) a n d g i v e s a c r u d e id e a o f h o w

c o n t e x t c a n e f f e c t p e r c e p t i o n i n o u r m o d e l s . N o t e t h a t i n F i g u r e 8 w e a r e

u si ng a s h o r t h a n d n o t a t i o n . A m o d i f i e r t o u c h i n g a d o u b l e - e n d e d a r r o w

a c t u a l ly b l o c k s t w o c o n n e c t i o n s . S o m e t i m e s w e a ls o o m i t t h e a r r o w h e a d s

w h e n c o n n e c t i o n i s d o u b l e - e n d e d . )

M a p p i n g s c a n a l so b e u s ed t o s el ec t a m o n g a n u m b e r o f p o s s i b le

v a lu e s . C o n s i d e r t h e e x a m p l e o f t h e r e l a t i o n b e t w e e n d e p t h , p h y s i ca l si ze ,

a n d r e t i n a l si ze o f a ci rc l e. F o r n o w , a s s u m e t h a t t h e c i r cl e is c e n t e r e d o n

a n d o r t h o g o n a l t o t h e l i n e o f s i g h t, t h a t t h e f o c u s is f i x e d , e t c . ) T h e n t h e r e i s

a f i x e d r e l a t i o n b e t w e e n t h e s i ze o f r e t i n a l i m a g e a n d t h e s iz e o f t h e p h y s i c a l

c i r c l e f o r a n y g i v e n d e p t h . T h a t i s , e a c h d e p t h s p e c i f i e s a m pping f r o m

r e t in a l t o p h y s i c a l s iz e s e e F i g . 9) . H e r e w e s u p p o s e t h e s c al es f o r d e p t h a n d

t h e t w o s iz es a r e c h o s e n s o t h a t u n i t d e p t h m e a n s t h e s a m e n u m e r i c a l s iz e . I f

w e k n e w t h e d e p t h o f t h e o b j e c t b y t o u c h , c o n t e x t , o r m a g i c ) w e w o u l d

k n o w i t s p h y s i c a l s i z e . T h e n e t w o r k a b o v e a l l o w s r e t i n a l s i z e 2 t o r e i n f o r c e

p h y s i c a l s iz e 2 w h e n d e p t h = 1 b u t i n h i b i t s t h i s c o n n e c t i o n f o r a ll o t h e r

d e p t h s . S i m i l a r l y , a t d e p t h 3 , w e s h o u l d i n t e r p r e t r e t i n a l si ze 2 as p h y s i c a l

s iz e 8 , a n d i n h i b i t o t h e r i n t e r p r e t a t i o n s . S e v e r a l r e m a r k s a r e in o r d e r . F i r s t ,

n o t i c e t h a t t h is n e t w o r k i m p l e m e n t s a f u n c t i o n p h y s = f r e t , d e p ) th a t m a p s

f r o m r e t in a l s iz e a n d d e p t h t o p h y s i c a l s iz e, p r o v i d i n g a n e x a m p l e o f h o w t o

r e p l a c e f u n c t i o n s w i t h p a r a m e t e r s b y m a p p i n g s . F o r t h e s i m p l e c a s e o f

l o o k i n g a t o n e o b j e c t p e r p e n d i c u l a r t o t h e l i n e o f s i g h t , t h e r e w i l l b e o n e

c o n s i s t e n t c o a l i t i o n o f u n i t s w h i c h w i l l b e s t a b l e . T h e w o r k d o e s s o m e t h i n g

m o r e , a n d t h i s i s c r u c i a l t o o u r e n t e r p r i s e ; t h e n e t w o r k c a n r e p r e s e n t t h e

c o n s i s t e n c y r e l a t i o n R a m o n g t h e t h r e e q u a n t i t i e s : d e p t h , r e t i n a l s i z e , a n d


20/50

4 FELDMAN AND BALLARD

p h y s i c a l si ze . I t e m b o d i e s n o t o n l y t h e f u n c t i o n f , b u t i t s t w o i n v e r se f u n c -

t i o n s a s w e ll (d e p = f ~ ( r e t , p h y s ) , a n d r e t = f 2 ( p h y s , d e p ) ) . ( T h e n e t w o r k a s

s h o w n d o e s n o t i n c l u d e t h e li n k s f o r f , a n d f~ , b u t t h e s e a r e s i m i l a r t o t h o s e

f o r f . ) M o s t o f S e c t i o n 5 i s d e v o t e d t o l a y i n g o u t n e t w o r k s t h a t e m b o d y

t h e o r i e s o f p a r t i c u l a r v i s u a l c o n s i s t e n c y r e l a t i o n s .

T h e i d e a o f m o d i f i e r s is , in a se n se , c o m p l e m e n t a r y t o t h a t o f c o n -

j u n c t i v e c o n n e c t i o n s . F o r e x a m p l e , t h e n e t w o r k o f F i g u r e 9 c o u l d b e t r an s -

f o r m e d i n t o t h e f o l l o w i n g n e t w o r k ( F ig . 10 ). I n th i s n e t w o r k t h e v a r ia b l e s

f o r p h y s i c a l s iz e , d e p t h , a n d r e t i n a l s iz e a r e a l l g i v e n e q u a l w e i g h t . F o r e x -

a m p l e , p h y s i c a l si ze = 4 a n d d e p t h = 1 m a k e a conjunctive connection w i t h

r e ti n a l s iz e = 4 . E a c h o f t h e v a l u e u n i ts in a c o m p e t i n g r o w c o u l d b e c o n -

n e c t e d t o a ll o f i ts c o m p e t i t o r s b y i n h i b i t o r y l i n ks a n d t h is w o u l d t e n d t o

m a k e t h e n e t w o r k a c t i v a t e o n l y o n e v a l u e i n e a c h c a t e g o r y . T h e g e n e r a l

i ss u e o f r i v a l r y a n d c o a l i t i o n s w ill b e d i sc u s s e d i n t h e n e x t t w o s u b - s e c t i o n s .

W h e n s h o u l d a r e l a t i o n b e i m p l e m e n t e d w i t h m o d i f i e r s a n d w h e n

s h o u l d i t b e im p l e m e n t e d w i th c o n j u n c t i v e c o n n e c t i o n s ? A s i m p l e, n o n -

r i g o r o u s a n s w e r t o t hi s q u e s t i o n c a n b e o b t a i n e d b y e x a m i n i n g t h e s iz e o f

t w o s e t s o f u n i t s: (1 ) t h e n u m b e r o f u n it s t h a t w o u l d h a v e to b e in h i b i t e d b y

m o d i f i e r s ; a n d ( 2 ) t h e n u m b e r o f u n i t s t h a t w o u l d h a v e t o b e r e i n f o r c e d

w i t h c o n j u n c t i v e c o n n e c t i o n s . I f ( 1 ) i s l a r g e r t h a n ( 2 ) , t h e n o n e s h o u l d

c h o o s e m o d i f i e r s ; o t h e rw i s e c h o o s e c o n j u n c t i v e c o n n e c t i o n s . S o m e t i m e s

t h e c h o ic e is o b v i o u s : t o i m p l e m e n t th e b r o w n C a l i f o r n i a n g r a ss e x a m p l e o f

F i g u r e 8 w i th c o n j u n c t i v e c o n n e c t i o n s , o n e w o u l d h a v e t o r e i n f o r c e al l u n it s

r e p r e s e n t i n g p l a c e s t h a t h a d g r e e n g r a s s C l e a r l y i n t h i s c a s e i t is e a s i e r t o

h a n d l e t h e e x c e p t io n w i t h m o d i f i e r s . O n t h e o t h e r h a n d , t h e d e p t h r e l a ti o n

R ( p h y , d e p , r e t ) i s m o r e c h e a p l y i m p l e m e n t e d w i t h c o n j u n c t i v e c o n n e c t i o n s .

S i n c e o u r m o d i f i e r s a r e s t r i c t l y b i n a r y , c o n j u n c t i v e c o n n e c t i o n s h a v e t h e

a d d i ti o n a l a d v a n t a g e o f c o n t i n u o u s m o d u l a t i o n .

T o s ee h o w t h e c o n j u n c t i v e c o n n e c t i o n s t r a te g y w o r k s i n g e n e r a l, s u p -

p o s e a c o n s t r a i n t r e l a t i o n t o b e s a t is f i e d in v o l v e s a v a r i a b l e x , e . g . , f ( x , y , z , w )

= 0 . F o r a p a r t i c u l a r v a l u e o f x , t h e r e w i ll b e tr i p le s o f v a l u e s o f y , z , a n d w

t h a t s a t i s fy th e r e l a t i o n f . E a c h o f th e s e tr i p le s s h o u l d m a k e a c o n j u n c t i v e

c o n n e c t i o n w i t h t h e u n i t r e p r e s e n t i n g t h e x - v a l u e . T h e r e c o u l d a l s o b e 3 - i n -

p u t c o n j u n c t i o n s a t ea c h v a l u e o f y , z , w . E a c h o f t he s e f o u r d i f f e r e n t k in d s

o f c o n j u n c t i v e c o n n e c t i o n s c o r r e s p o n d s t o a n i n t e r p r e t a t i o n o f t h e relation

f ( x , y , z ,w ) = 0 a s a function i . e . , x = f , ( y , z , w ) , y = f2 (x , z , w) , z = f3 (x , y , w ) , o r

w = f , ( x ,y , z ) . O f c o u r s e , t h e s e f u n c t i o n s n e e d n o t b e s i n g le - v a l u e d . T h i s n e t -

w o r k c o n n e c t i o n p a t t e r n c o u l d b e e x t e n d e d t o m o r e t h a n f o u r v a ri a b le s , b u t

h i g h n u m b e r s o f v a r i a b l e s w o u l d t e n d t o i n c r e a s e i ts s e n s it iv i t y t o n o i s y in -

p u t s . H i n t o n h a s s u g g e s te d a s p e c ia l n o t a t i o n f o r th e s i t u a t io n w h e r e a n e t-

w o r k e x a c t l y c a p t u r e s a c o n s i s t e n c y r e la t i o n . T h e m u t u a l l y c o n s i s t e n t v a l u e s

a r e a ll s h o w n t o b e c e n t r a l l y l i n k e d ( F i g . 1 1). T h i s n o t a t i o n p r o v i d e s a n e le -


21/50

a

u_ e ~

e -

a . .

1 3

0

c :

0

z

e

Q .

a~

14.

m

5


22/50

0

u ~

~ . ~ . ~

i

~ ~

r ~

o

r

t

o

u

o

>

c

0

U

. ~

0

0

Z

n

2 2 6


23/50

CONNECTIONIST MODELS A N D THEIR PROPERTIES

F ig u r e 1 N o t a t i o n f o r c o n s i s te n c y r e l a t i o n s

7

g a n t w a y o f p r e s e n ti n g th e i n t e r a c t i o n s a m o n g n e t w o r k s , b u t m u s t b e u s e d

w i th c a r e . W r i t i n g d o w n a t r ia n g l e d i a g r a m d o e s n o t i n su r e t h a t t h e u n d e r -

l y in g m a p p i n g s c a n b e m a d e c o n s i s t e n t o r c o m p u t a t i o n a l l y w e l l - b e h a v e d .

W i n n er T a k e A l l N e t w o r k s a n d R e g u l a te d N e t w o r k s

A v e r y g e n e r a l p r o b l e m t h a t a ri se s in a n y d i s t r i b u t e d c o m p u t i n g s i t u a t io n is

h o w t o g e t t h e e n t i re s y s te m t o m a k e a d e c is i o n ( o r p e r f o r m a c o h e r e n t ac -

t i o n , e t c .) . B i o lo g i c a ll y n e c e s s a r y e x a m p l e s o f t h is b e h a v i o r a b o u n d ; r a n g i n g

f r o m t u r n i n g l e ft o r ri g h t , t h r o u g h f i g h t -o r - f li g h t r e s p o n s e s, t o i n t e r p r e t a -

t i o n s o f a m b i g u o u s w o r d s a n d i m a g e s . D e c i s i o n - m a k i n g is a p a r t i c u l a r l y i m -

p o r t a n t i ss ue f o r t h e c u r r e n t m o d e l b e c a u s e o f it s r e s t ri c t io n s o n i n f o r m a t i o n

f l o w a n d b e c a u s e o f t h e a l m o s t l i n ea r n a t u r e o f t h e p - u n i t s u s e d i n m a n y o f

o u r s p e c i fi c e x a m p l e s . D e c i s i o n - m a k i n g i n t r o d u c e s t h e n o t i o n s o f

stable

states a n d convergence o f n e t w o r k s .

O n e w a y t o d e a l w i t h t h e is su e o f c o h e r e n t d e c is i o ns i n a c o n n e c t i o n i s t

f r a m e w o r k is t o i n t r o d u c e winner-take-all ( W T A ) n e t w o r k s , w h i c h h a v e t h e

p r o p e r t y t h a t o n l y t h e u n i t w i th t h e h i g h es t p o t e n t i a l ( a m o n g a se t o f c o n -

t e n d e r s ) w i ll h a v e o u t p u t a b o v e z e r o a f t e r s o m e s e t t in g t i m e ( F ig . 1 2) . T h e r e

a r e a n u m b e r o f w a ys to c o n s t r u c t W T A n e t w o r k s f r o m t h e u n i ts d e s c ri b e d

a b o v e . F o r o u r p u r p o s e s i t is e n o u g h t o c o n s i d e r o n e e x a m p l e o f a W T A n e t-

w o r k w h i c h w i ll o p e r a t e i n o n e t i m e s t e p f o r a s et o f c o n t e n d e r s e a c h o f

w h o m c a n r e a d t h e p o t e n t i a l o f a l l o f t h e o t h e r s . E a c h u n i t i n t h e n e t w o r k

c o m p u t e s i t s n e w p o t e n t i a l a c c o r d i n g t o t h e r u l e :

p- -i f p >max ij, .1) then else O.


24/50

8 F E L D M A N A N D B A L LA R D

T h a t i s , e a c h u n i t s e t s i t s e l f t o z e r o i f i t k n o w s o f a h i g h e r i n p u t . T h i s is f a s t

a n d s i m p l e , b u t p r o b a b l y a li tt le t o o c o m p l e x t o b e p la u s i b l e as th e b e h a v i o r

o f a s in g le n e u r o n . T h e r e i s a s t a n d a r d t r ic k ( a p p a r e n t l y w i d e ly u s e d b y

n a t u r e ) t o c o n v e r t t hi s in t o a m o r e p l a u s i b l e s c h e m e . R e p l a c e e a c h u n i t

a b o v e w i th t w o u n it s; o n e c o m p u t e s t h e m a x i m u m o f th e c o m p e t i t o r s in -

p u t s a n d i n h ib i ts t h e o t h e r . T h e c i r cu i t a b o v e c a n b e s tr e n g t h e n e d b y a d d i n g

a re v e r s e i n h i b i t o r y l in k , o r o n e c o u l d u s e a m o d i f i e r o n t h e o u t p u t , e t c . O b -

v i o u s ly o n e c o u l d h a v e a W T A l a y e r t h a t g o t i n p u ts f r o m s o m e s et o f c o m -

p e t i to r s a n d s e tt le d t o a w i n n e r w h e n t r i g g e r e d to d o s o b y s o m e d o w n s t r e a m

n e t w o r k . T h i s is a n e x a c t a n a l o g y o f s t r o b i n g a n o u t p u t b u f f e r in a c o n v e n -

t i o n a l c o m p u t e r .

O n e p r o b l e m w i t h p r e v i o u s n e u r a l m o d e l i n g a t t e m p t s i s t h a t t h e ci r-

c u it s p r o p o s e d w e r e o f t e n u n n a t u r a l l y d e l ic a te ( u n s t a b le ) . S m a l l c h a n g e s i n

p a r a m e t e r v a l u e s w o u l d c a u s e t h e n e t w o r k s t o o s c i l la t e o r c o n v e r g e to i n c o r -

r e c t a n s w e r s . W e w ill h a v e t o b e c a r e f u l n o t t o f a ll i n t o t h i s tr a p , b u t w o u l d

l ik e t o a v o i d d e t a i le d a n a l y s i s o f e a c h p a r t i c u l a r m o d e l f o r d e l i c a c y in t h is

p a p e r . W h a t a p p e a r s t o b e r e q u ir e d a r e s o m e b u il d in g b l o c k s a n d c o m b i n a -

t io n r u l es th a t p r e s e r v e th e d e s ir e d p r o p e r t i e s . F o r e x a m p l e , t h e W T A s u b -

n e t w o r k s o f th e l as t e x a m p l e w i ll n o t o s c i l la t e in t h e a b s e n c e o f o s c i ll a t in g

i n p u ts . T h i s is a l so t ru e o f a n y s y m m e t r i c m u t u a l l y i n h i b i t o r y s u b n e t w o r k .

T h i s is i n tu i ti v e ly c l e a r a n d c o u l d b e p r o v e n r i g o r o u s l y u n d e r a v a r i e t y o f

a s s u m p t i o n s ( cf . G r o s s b e r g , 1 98 0). I f e v e r y u n i t r ec e iv e s i n h i b i ti o n p r o p o r -

t i o n a l t o t h e a c t i v i t y ( p o t e n t i a l ) o f e a c h o f it s r i v a ls , t h e i n s t a n t a n e o u s

l e a d e r w i ll r e c e i v e le s s i n h i b i t i o n a n d t h u s n o t l o s e i ts l e a d u n l e s s t h e i n p u t s

c h a n g e s i g n i f i c a n t l y .

A n o t h e r u s e fu l p ri n c ip l e is t h e e m p l o y m e n t o f l o w e r - b o u n d a n d u p p e r -

b o u n d c e ll s t o k e e p t h e t o t a l a c t iv i t y o f a n e t w o r k w i t h i n b o u n d s ( F ig . 13).

S u p p o s e t h a t w e a d d t w o e x t r a u n i ts , L B a n d U B , t o a n e t w o r k w h i c h h a s

c o o r d i n a t e d o u t p u t . T h e L B c e ll c o m p a r e s t h e t o t a l ( s u m ) a c t i v i ty o f th e

u n i ts o f th e n e t w o r k w i t h a l o w e r b o u n d a n d s e n d s p o s i t i v e a c t i v a t i o n u n i -

f o r m l y t o a l l m e m b e r s i f t h e s u m is t o o l o w . T h e U B c el l i n h i b i ts a l l u n i t s

e q u a l l y i f t h e s u m o f a c t i v i t y is t o o h i g h . N o t i c e t h a t L B a n d U B c a n b e

p a r a m e t e r s s e t f r o m o u t si d e t h e n e t w o r k . U n d e r a w id e r a n g e o f co n d i t io n s

( b u t n o t al l ) , t h e L B - U B a u g m e n t e d n e t w o r k c a n b e d e s i g n e d to p r e s e r v e

o r d e r r e l a t i o n s h i p s a m o n g t h e o u t p u t s v j o f th e o r ig i n a l n e t w o r k w h i le k e e p -

in g t h e s u m b e t w e e n L B a n d U B .

W e w ill o f te n a s s u m e t h a t L B - U B p a i r s a r e u se d t o k e e p th e s u m o f

o u t p u t s f r o m a n e t w o r k w i th in a g iv e n r a n g e . T h i s s a m e m e c h a n i s m a ls o

g o e s f a r t o w a r d s e l i m i n a t in g t h e tw i n p e ri ls o f u n i f o r m s a t u r a t i o n a n d

u n i f o r m s il en c e w h i c h c a n e a s il y a r is e in m u t u a l i n h i b i t io n n e t w o r k s . T h u s

w e will o f t e n b e a b le t o r e a s o n a b o u t t he c o m p u t a t i o n o f a n e t w o r k a s s u m -

i n g t h a t i t s t a y s a c t i v e a n d b o u n d e d .


25/50

a

-i

a

t-

C~

,.C

0

c ~

L~

C~

_ o

r--

L

O

0~

C

O ~

Z aJ

c ~

-j

0

LL

2 ~ 9


26/50

2 3 F E L D M A N A N D B A LL AR D

S t a b l e o a l i t i o n s

F o r a m a s s i v e l y p a r a l l e l s y s t e m to a c t u a l l y m a k e a d e c i s io n ( o r d o s o m e -

t h i n g ), t h e r e w ill h a v e t o b e s ta t e s in w h i c h s o m e a c t i v i t y s t r o n g l y d o m i n a t e s .

S u c h s t a bl e , c o n n e c t e d , h i g h c o n f i d e n c e u n i ts a r e t e r m e d s t a b l e c o a l i t i o n s .

A s t a b l e c o a l i ti o n i s o u r a r c h i t e c t u r a l ly - b i a s e d t e r m f o r t h e p s y c h o l o g i c a l

n o t i o n s o f p e r c e p t , a c t io n , e t c . W e h a v e s h o w n s o m e s i m p l e i n s ta n c e s o f

s t a b le c o a l i t i o n s , i n F i g u r e 5 b a n d t h e W T A n e t w o r k . I n t h e d e p t h n e t w o r k s

o f F i g u r e s 9 a n d 10, a s t a b l e c o a l i t i o n w o u l d b e t h r e e u n i t s r e p r e s e n t i n g c o n -

s i s te n t v a l u e s o f r e t in a l s i z e , d e p t h , a n d p h y s i c a l s iz e . B u t t h e g e n e r a l i d e a is

t h a t a v e r y l a r g e c o m p l e x s u b s y s t e m m u s t s t a b i l iz e , e . g . , t o a f i x e d i n t e r p r e -

t a t i o n o f v is u a l i n p u t , a s i n F i g u r e I . T h e w a y w e b e li e v e t h i s t o h a p p e n is

t h r o u g h m u t u a l l y r e i n f o r c i n g c o a l i ti o n s w h i c h d o m i n a t e a ll r iv a l a c t iv i t y

w h e n t h e d e c i s io n is r e q u i r e d . T h e s i m p l e s t c a s e o f t h is is F i g u r e 5 b , w h e r e

t h e t w o u n it s A a n d B f o r m a c o a l i t i o n w h i c h s u p p r e s s e s C a n d D . F o r m a l l y ,

a c o a l it io n w i ll b e c a ll e d s t a b l e w h e n t h e o u t p u t o f a ll i ts m e m b e r s i s n o n

d e c re a s i n g . N o t i c e t h a t a c o a l i t i o n is n o t a p a r t i c u l a r a n a t o m i c a l s t r u c t u r e ,

b u t a n i n s t a n t a n e o u s l y m u t u a l l y r e i n f o r c i n g s et o f u n i ts , in

Connectionist models and their properties.pdf

Documents

Transcript of Connectionist models and their properties.pdf