Activation in Human MTMST by Static Imageswith Implied Motion

Zoe Kourtzi and Nancy KanwisherMassachusetts Institute of Technology

Abstract

amp A still photograph of an object in motion may conveydynamic information about the position of the objectimmediately before and after the photograph was taken(implied motion) Medial temporalmedial superior temporalcortex (MTMST) is one of the main brain regions engaged inthe perceptual analysis of visual motion In two experimentswe examined whether MTMST is also involved in representing

implied motion from static images We found strongerfunctional magnetic resonance imaging (fMRI) activation with-in MTMST during viewing of static photographs with impliedmotion compared to viewing of photographs without impliedmotion These results suggest that brain regions involved inthe visual analysis of motion are also engaged in processingimplied dynamic information from static images amp

The perception of motion is critical for our ability tointeract with a dynamic environment Neurophysiolo-gical studies in monkeys (for example Britten News-ome Shalden Celebrini amp Movshon 1996 Dubner ampZeki 1971 Maunsell amp Van Essen 1983 Van EssenMaunsell amp Bixby 1981) and imaging studies in hu-mans (Dupont Orban De Bruyn Verbruggen amp Mor-telmans 1994 Tootell et al 1995b Watson et al1993 Zeki et al 1991) have shown that a networkof brain regions in the primate visual system is devotedto the important task of analyzing visual motion Oneof the main regions involved in motion processing isthe extrastriate visual area medial temporalmedialsuperior temporal cortex (MTMST) Recent imagingstudies have shown that MTMST is involved not onlyin the analysis of the continuous coherent motion of aphysical stimulus but also in the processing of appar-ent motion (Goebel Khorram-Sefat Muckli Hacker ampSinger 1998 Kaneoke Bundou Koyama Suzuki ampKakigi 1997) illusory motion (Tootell et al 1995aZeki Watson amp Frackowiak 1993) and imagined mo-tion (Goebel et al 1998 OrsquoCraven amp Kanwisher1997)

Most physiological and imaging studies of MTMSThave used stimuli such as moving dots and gratingsThese stimuli consist of multiple sequential frames eachof which contains information about the position of thestimulus in space at a specific moment in time Howeverin naturally occurring motion an instantaneous framefrom a continuous-motion sequence often contains in-formation not only about the current position of theobjects in the frame but also about their motion trajec-tory Based on our knowledge of how animate and

inanimate objects move we can infer the position ofobjects in a subsequent moment in time Consider thelsquolsquoaction photographrsquorsquo in Figure 2a The motion impliedin this photograph allows us to anticipate the futureposition of the actor a moment after the photographwas taken Psychophysical studies have demonstratedthat observers extract this kind of dynamic informationby extrapolating an objectrsquos future position from themotion implied in a static photograph Specificallywhen asked to judge whether two still photographsare the same or different subjects often wrongly cate-gorize them as identical when the second one is aphotograph of the same event depicted in the firstphotograph but taken a moment later in time (Freyd1983) These studies suggest that dynamic informationcan be extracted from still photographs even when thetask does not require it

The current studies were designed to test whetherbrain areas known to be involved in the analysis ofphysical stimulus motion are also engaged in proces-sing dynamic information from static images withimplied motion To this end we used functionalmagnetic resonance imaging (fMRI) to localize areaMTMST in each subject individually and then mea-sured activity in this area while the subjects observedstatic photographs of human athletes in action (im-plied motion images) or of athletes at rest (no impliedmotion) In two further conditions in the same scanssubjects viewed another set of photographs of houses(an example of a stimulus conveying no dynamicinformation) and photographs of people at rest (tocontrol for the possibility that the athletes at restcould be associated with information about action

copy 2000 Massachusetts Institute of Technology Journal of Cognitive Neuroscience 121 pp 48ndash55

since athletes were also presented the implied motioncondition) Half the subjects viewed these four differ-ent kinds of photographs passively To ensure atten-tion to stimuli from all conditions the other half ofthe subjects performed a lsquolsquo1-backrsquorsquo repetition detectiontask on the same sequences In a second experimentwe tested the response of area MTMST to photo-

graphs of animals and nature scenes that either de-picted implied motion or did not

RESULTS

The localizer scans (low contrast moving vs stationaryrings) successfully localized each subjectrsquos MTMST in

Figure 1 Functional data are overlaid on a high-resolution T1-weighted anatomical image for each slice Right hemisphere appears on the leftSignificance levels reflect the results of t-tests on the MR signal intensity ( plt10ndash7 equivalent to plt10ndash1 after Bonferroni correction) I Groupanalysis on functional data from 5 subjects (coregistered in Talairach space) showing regions responding significantly to (a) moving vs stationaryrings and (b) images with implied motion vs images without implied motion (Experiment 1) The green circles indicate regions activated significantlyfor both moving vs stationary rings and images with implied motion vs images without implied motion II Five slices from one subject showingactivation for viewing of (a) moving vs stationary rings and (b) images with implied motion vs images without implied motion (Experiment 1)

Kourtzi and Kanwisher 49

the lateral occipital region (Figure 1) consistent withprior reports (for example Tootell et al 1995b) Foreach subject this region served as the region of interest(ROI) from which the response was extracted for eachof the experimental conditions for the same subjectThe response for each condition and subject was quan-tified as the percent signal change (PSC) from thefixation baseline condition The average PSC acrosssubjects for each condition and the time course ofsignal intensity averaged across subjects are shown inFigure 2 for Experiment 1 and Figure 3 for Experiment2

For the first experiment a two-way ANOVA (StimulusTypepoundTask) on the PSC for each condition across sub-jects with Stimulus Type (implied motion athletes noimplied motion athletes people at rest houses) as thewithin-subjects variable and Task (passive 1-back) as thebetween-subjects variable showed a significant maineffect of Stimulus Type (F(3 18)=201 plt001) There

was no main effect of Task (F(1 18)lt1) and no inter-action of Stimulus Type and Task (F(3 18)=12 pgt3)The PSC in MTMST was significantly greater for imagesof athletes with implied motion vs athletes withoutimplied motion in both the passive (t(3)=35 plt05)and the 1-back (t(3)=45 plt05) tasks The PSC in MTMST during viewing of athletes without implied motionwas not significantly different from that for people atrest (t(7 )=06 pgt5)

The similar patterns of activation in MTMST acrosspassive viewing and 1-back tasks suggest that the ob-served activation is not likely to be due to differences intask difficulty or attentional allocation across conditionsIndeed the behavioral data from the 1-back task suggestthat this matching task was at least as difficult for imageswithout implied motion as for images with impliedmotion Specifically across three out of the four subjects(the behavioral data for one subject were lost due to acomputer error) the average percent correct detection

Figure 2 Results of Experiment 1 (a) An example stimulus from each condition Average percent signal increase (from the fixation baseline) andstandard deviations across subjects for each stimulus type in MT observed for each task (passive viewing 1-back) as well as the average across tasks(b) The time course of the percent change in MR signal intensity (from the fixation baseline) in MT over the period of the scan Black dot indicatesfixation IM images of athletes with implied motion no-IM images of athletes without implied motion R people at rest H houses

50 Journal of Cognitive Neuroscience Volume 12 Number 1

and total number of false alarms (in parentheses) overfour epochs for each condition were Implied motion90 (1) no implied motion 82 (1) people at rest93 (2) and Houses 74 (1)

The data from Experiment 2 were analyzed by a two-way repeated ANOVA (ConditionpoundStimulus Type) withCondition (implied motion vs no implied motion) andStimulus Type (animals vs nature scenes) as repeatedmeasures variables A main effect of Condition (F(13)=2685 plt001) was observed No main effect ofStimulus Type (F(1 3)=6847 p=079) nor a significantinteraction of Stimulus Type and Condition (F(1 3)lt1)was observed The PSC within MTMST was significantlygreater for implied than for no implied motion condi-tions for both animals (t(3)=37 plt05) and naturescenes (t(3)=61 plt01)

In order to look at regions of the brain beyond MTMST KolmogorovndashSmirnov statistics were run on eachvoxel scanned in each subject in Experiment 1 testingwhether that voxel showed stronger activation for (i)moving vs stationary rings in the localizer runs and (ii)implied motion athletes vs no implied motion athletes

For all subjects the lateral occipital regions thatshowed significant activation for moving vs stationaryrings in the localizer task overlapped with regionsshowing significant activation for implied motion vsno implied motion in the experimental runs Howeverfor all subjects the implied motion vs no impliedmotion athletes comparison also activated other re-gions contiguous to MTMST extending medially ante-riorly and posteriorly Six out of the eight subjects inExperiment 1 also showed significant activation forimplied motion vs no implied motion in the regionof the superior temporal sulcus Activations in theseregions were also observed in t-test group analyses offive subjects coregistered into Talairach space (Talair-ach amp Tournoux 1988) (Three subjects could not becoregistered due to poor resolution in the frontalregions of the brain as a result of surface coil usage)These analyses (see Figure 1) showed significantlystronger activations ( plt10ndash7 equivalent to plt10ndash1

after Bonferroni correction) for moving compared tostatic rings in MTMST and for implied motion vs noimplied motion

Figure 3 Results of Experiment 2 (a) An example stimulus from each condition Average percent signal change and standard deviations acrosssubjects for each stimulus type in MT ( b) The time course of the percent change in MR signal intensity in MT over the period of the scan Black dotindicates fixation AIM images of animals with implied motion Ano IM images of animals without implied motion SIM images of nature sceneswith implied motion Sno IM images of nature scenes without implied motion

Kourtzi and Kanwisher 51

DISCUSSION

Our results suggest that cortical areas involved in theanalysis of physical stimulus motion can be also en-gaged automatically by static images that merely implymotion Specifically passively observed static snapshotsof objects in action activate human motion areas (MTMST) more than static images of objects withoutimplied motion These results are observed for imagesimplying animate motion such as humans or animalsin action as well as inanimate motion such as activenature scenes

It is unlikely that these results can be explained bylow-level differences among the images in the differentconditions (for example differences in the location ofthe luminance edges) The activation in MTMST wassystematically greater for implied than no implied mo-tion across eight very different stimulus categories usedin the two experiments Furthermore it is unlikely thatthe modulation of activity in MT MST is related todifferences in image flicker (each photograph was dis-played for 300 msec followed by a 500-msec blankinterval followed by the next stimulus) since this flickeroccurred in all of our stimulus conditions

These results raise numerous questions about theanalysis of object motion in the human brain That isis MTMST involved in extracting implied motion infor-mation or is it influenced by such processes occurringelsewhere in the brain It seems unlikely that theperceptual analyses involved in the inference of motionfrom still images could be computed within MTMSTNeurophysiological and imaging studies have stronglysupported the role of MTMST in the analysis of stimulusmotion but not in processes such as object recognitionInferring motion from still images depends on objectcategorization and knowledge about the repertoire ofbehavior different objects can exhibit It seems mostlikely that such high-level perceptual inferences occurelsewhere in the brain and modulate activity in MTMSTin a top-down fashion Thus the observed activationsmay reflect an expectancy of object motion that could berepresented or influence representations in areas in-volved in processing physical stimulus motion (that isMTMST)

Consistent with this hypothesis the activation forimplied vs no implied motion extended beyond MTMST to several contiguous regions as shown in Figure 1These results are consistent with recent studies suggest-ing that other areas extending posterior and superior oranterior and inferior to MTMST are also involved inmotion analysis (De Jong Shipp Skidmore Frackowiakamp Zeki 1994 Dupont et al 1994 Shipp De Jong ZihlFrackowiak amp Zeki 1994 Watson et al 1993 ) Previousresearch has shown activation anterior and medial to MTfor passive viewing of images of illusory motion (ZekiWatson amp Frackowiak 1993) tool naming (MartinWiggs Ungerleider amp Haxby 1996) and the genera-

tion of action words (Martin Haxby Lalonde Wiggs ampUngerleider 1995) Recent imaging studies have shownactivation for motion boundaries in areas V3A (Tootellet al 1997 ) and KO (Orban Dupont De Bruyn VogelsVandenberghe amp Mortelmans 1995 Van OostendeSunaert Van Hecke Marchal amp Orban 1997 ) extendingposterior and medial to MT along the occipital surfaceThe activations observed in our subjects in the vicinity ofthe superior temporal sulcus are also consistent withprevious studies showing activation in the superiortemporal sulcus for motion imagery (Goebel et al1998) and viewing of biological motion stimuli (BondaPetrides Ostry amp Evans 1996 Puce Allison BentinGore amp McCarthy 1998)

Finally several prior findings support the hypothesisthat the current results reflect top-down influences ofhigh-level perceptual inferences on MTMST Both sin-gle unit (Treue amp Maunsell 1996 ) and fMRI studies(Beauchamp Cox amp DeYoe 1997 Corbetta MiezinDobmeyer Shulman amp Petersen 1990 1991 OrsquoCravenRosen Kwong Treisman amp Savoy 1997) have demon-strated that the response of MTMST to moving stimulican be strongly modulated by visual attention Alsoactivity in MTMST has been demonstrated even whensubjects close their eyes and merely imagine movingcompared to stationary arrays (Goebel et al 1998OrsquoCraven amp Kanwisher 1997 )

While the present work is consistent with these pre-vious studies suggesting that activation in MTMST canbe modulated in a top-down fashion we show here forthe first time that such top-down effects can occurautomatically That is dynamic information implicit inthe image was extracted and influenced activity in MTMST even though subjects were not asked or requiredto perceive attend to or imagine motion

One possible interpretation of our findings is thatinferring motion may involve or result in motion ima-gery Another interpretation is that the processing of aparticular object category (for example animals) maylead to activation of regions involved in processingproperties highly associated with that object category(for example motion) (Chao Haxby Lalonde Ungerlei-der amp Martin 1998 Martin et al 1996) Consistent withthe second hypothesis the current findings show thatactivation in MTMST is significantly higher for images ofpeople even people at rest than for images of houses

More broadly the current results support an emer-ging view of extrastriate cortex as playing a crucial rolenot only in visual perception but also in visual cogni-tion

METHODS

Subjects

Ten right-handed MIT students participated in Experi-ment 1 four in the passive viewing condition and six inthe 1-back matching condition Two subjects tested on

52 Journal of Cognitive Neuroscience Volume 12 Number 1

the 1-back matching condition were excluded from theanalysis due to excessive head motion Another six right-handed MIT students participated in Experiment 2 Twosubjects were excluded from the analysis in this condi-tion due to excessive head motion

Materials and Design

The stimuli used for functionally localizing MT were lowcontrast moving vs stationary concentric rings as de-scribed in Tootell et al (1995a) For the experimentalconditions stimuli were 300pound300 pixel digitized grays-cale photographs Experiment 1 involved a mixed de-sign with Stimulus Type a within-subject variable (withfour levels photographs of athletes with implied mo-tion athletes without implied motion people at restand houses) and Task a between-subjects factor (withtwo levels passive viewing vs 1-back repetition detec-tion) Experiment 2 involved two orthogonal factorscrossed within subjects Stimulus Type (animals vsscenes) and Condition (implied motion vs no impliedmotion)

Procedure

Each subject was run on two or more functional MTlocalizer scans with low contrast moving vs stationaryconcentric rings (as described in Tootell et al 1995a)Then each subject was run on four scans of the experi-mental test materials For the passive viewing condi-tions the subjects were asked to observe the imagescarefully while fixating a dot in the center of the image(Monitoring of eye movements outside the scanner forthree subjects that participated in Experiment 1 andthree subjects that participated in Experiment 2 showedthat the number of eye movements was very small in allconditions and did not differ significantly across condi-tions) For the 1-back matching condition subjects wereinstructed to press a button whenever they saw twoidentical pictures in a row Two or more repetitionsoccurred in each epoch

Each scan lasted 5 min and 36 sec and consisted ofsixteen 16-sec epochs with fixation periods interleavedas shown in Figures 2 and 3 Twenty different photo-graphs of the same type were presented in each epochEach photograph was presented for 300 msec with ablank interval of 500 msec between photographs Eachof the four stimulus types in each experiment werepresented in four different epochs within each scan ina design that balanced for the order of conditions asshown in Figures 2 and 3

MRI Acquisition

Scanning was done on the 3 T scanner (modified byANMR for Echo Planar Imaging) at the MGH-NMRCenter in Charlestown MA A custom bilateral surface

coil (built by J Thomas Vaughan) provided a high signal-to-noise ratio in posterior brain regions A bite-bar wasused to minimize head motion Standard imaging pro-cedures (Gradient Echo pulse sequence TR 2 sec TE30 msec flip angle 908 1808 offset 25 msec) were usedas described previously (Tong Nakayama Vaughan ampKanwisher 1998) Twelve 6-mm-thick near-coronal sliceswere oriented parallel to the brainstem and covered theoccipital lobe as well as the posterior portions of thetemporal and the parietal lobes One hundred sixty-eight functional images were collected for each slice ineach scan

Data Analysis

Each subjectrsquos MTMST was identified from the aver-age of the functional localizer scans as the set of allcontiguous voxels in the vicinity of the ascending limbof the inferior temporal sulcus (Tootell et al 1995bWatson et al 1993 Zeki et al 1991) that showedsignificantly stronger activation to moving comparedto static low-contrast concentric rings on a Kolmogor-ovndashSmirnov test at the level of plt0001 (uncorrected)In principle significant differences in KolmogorovndashSmirnov statistics can reflect differences in the var-iance only rather than in the means across conditions(Aguirre Zarahn amp DrsquoEsposito 1998) However thefact that the region selected by this procedure didindeed respond more strongly during the movingthan stationary conditions was confirmed by subse-quent analyses In particular t-tests across subjectsrevealed that the percent signal change in the selectedROIs was higher for moving than stationary conditions(a difference of 09 (t(7)=73 plt001) for Experi-ment 1 and 07 (t(3)=63 plt01) for Experiment 2)Moreover as shown in Figure 1 t-tests on the aver-aged group data for five subjects showed significantlystronger activation to moving compared to static rings( plt10ndash7 equivalent to plt10ndash1 after Bonferronicorrection)

For the analysis of the experimental scans the timecourse of MR signal intensity was extracted from eachsubjectrsquos MTMST by averaging the data from allvoxels within the ROI The average percent signalchange in MTMST was calculated for each subjectand stimulus type using the average signal intensityduring fixation epochs for the same subject experi-ment and task as a baseline Because the fMRIresponse typically lags four to six seconds after theneural response our data-analysis procedure treatedthe first image in each epoch as belonging to thecondition of the preceding epoch and omitted thenext two images (during the transition betweenepochs) from the analysis

An ANOVA across subjects was run on the averagepercent signal change in each of the conditions in eachexperiment Because data were analysed within inde-

Kourtzi and Kanwisher 53

pendently defined ROIs for MTMST no correction formultiple voxelwise comparisons was required

Acknowledgments

We would like to thank Ted Adelson and Maggie Shiffrar fortheir helpful comments and suggestions on this project andBruce Rosen and many people at the MGH-NMR Center fortechnical assistance and support We would also like to thankPaul Downing and Russell Epstein for their comments onprevious versions of this manuscript This research wassupported by NIMH Grant 56037 and a Human Frontiersgrant to Nancy Kanwisher Some of the results discussed in thismanuscript were first presented at the 1998 meeting of theSociety for Neuroscience in Los Angeles CA and the 1998meeting of the Psychonomic Society in Dallas TX

Reprint requests should be sent to Zoe Kourtzi Dept of Brainand Cognitive Science MIT NE20-4043 77 Massachusetts AveCambridge MA 02139-4307 or via e-mail zoepsychemitedu

REFERENCES

Aguirre G K Zarahn E amp DrsquoEsposito M (1998) A critique ofthe use of the KolmogorovndashSmirnov (KS) statistic for theanalysis of BOLD fMRI data Magnetic Resonance in Medi-cine 39 500 ndash505

Beauchamp M S Cox R W amp DeYoe E A (1997) Gradedeffects of spatial and featural attention on human-area MTand associated motion-processing areas Journal of Neuro-physiology 78 516 ndash520

Bonda E Petrides M Ostry D amp Evans A (1996) Specificinvolvement of human-parietal systems and the amygdalain the perception of biological motion The Journal ofNeuroscience 16 3737ndash 3744

Britten K H Newsome W T Shalden M N Celebrini S ampMovshon J A (1996) A relationship between behavioralchoice and the visual responses of neurons in macaque MTVisual Neuroscience 13 87ndash100

Chao L L Haxby J V Lalonde F M Ungerleider L G ampMartin A (1998) Pictures of animals and tools deferen-tially engage object form-related and motion-related brainregions Paper presented at the 28th Annual Meeting of theSociety for Neuroscience LA CA

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1990) Attentional modulation of neuralprocessing of shape color and velocity in humans Science248 1556 ndash1559

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1991) Selective and divided attention duringvisual discriminations of shape color and speed Functionalanatomy by positron emission tomography Journal ofNeuroscience 11 2383ndash2402

De Jong B M Shipp S Skidmore B Frackowiak R S Jamp Zeki S (1994) The cerebral activity related to visualperception of forward motion in depth Brain 117 1039ndash1054

Dubner R amp Zeki S M (1971) Response properties andreceptive fields of cells in an anatomically-defined region ofthe superior temporal sulcus in the monkey Brain Re-search 35 528ndash532

Dupont P Orban G A De Bruyn B Verbruggen A ampMortelmans L (1994) Many areas in the human brain re-spond to visual motion Journal of Neurophysiology 721420ndash1424

Freyd J (1983) The mental representation of movement

when static stimuli are viewed Perception and Psycho-physics 33 575ndash581

Goebel R Khorram-Sefat D Muckli L Hacker H amp SingerW (1998) The constructive nature of vision Direct evidencefrom functional magnetic resonance imaging studies of ap-parent motion and motion imagery European Journal ofNeuroscience 10 1563 ndash1573

Kaneoke Y Bundou M Koyama S Suzuki H amp Kakigi R(1997) Human cortical area responding to stimuli in ap-parent motion NeuroReport 8 677ndash 682

Martin A Haxby J V Lalonde F M Wiggs C L amp Unger-leider L G (1995) Discrete cortical regions associated withknowledge of color and knowledge of action Science 270102 ndash105

Martin A Wiggs C L Ungerleider L G amp Haxby J V(1996) Neural correlates of category-specific knowledgeNature 379 649 ndash652

Maunsell J H amp Van Essen D C (1983) The connections ofthe middle temporal-visual area (MT) and their relationshipto a cortical hierarchy in the macaque monkey Journal ofNeuroscience 3 2563 ndash2586

OrsquoCraven K M amp Kanwisher N G (1997) Visual imagery ofmoving stimuli activates area MTMST Paper presented atthe 27th Annual Meeting of the Society for NeuroscienceNew Orleans LA

OrsquoCraven K M Rosen B R Kwong K K Treisman A ampSavoy R L (1997) Voluntary attention modulates fMRI ac-tivity in human MT-MST Neuron 18 591ndash 598

Orban G A Dupont P De Bruyn B Vogels R Vanden-berghe R amp Mortelmans L (1995) A motion area in humanvisual cortex Proceedings of the National Academy ofScience 92 993 ndash997

Puce A Allison T Bentin S Gore J C amp McCarthy G(1998) Temporal-cortex activation in human subjects view-ing eye and mouth movements Journal of Neuroscience18 2188 ndash2199

Shipp S de Jong B M Zihl J Frackowiak R S J amp ZekiS (1994) The brain activity related to residual motion vi-sion in a patient with bilateral lesions of V5 Brain 1171023 ndash1038

Talairach J amp Tournoux P (1988) Co-planar stereotaxicatlas of the human brain New York Thieme Medical

Tong F Nakayama K Vaughan J T amp Kanwisher N (1998)Binocular rivalry and visual awareness in human extrastriatecortex Neuron 21 753 ndash759

Tootell R B H Mendola J D Hadjikhani N K Ledden PJ Liu A K Reppas J B Sereno M I amp Dale A M(1997) Functional analysis of V3A and related areas in hu-man visual cortex The Journal of Neuroscience 15 7060 ndash7078

Tootell R B H Reppas J B Dale A M Look R B SerenoM I Malach R Brady T J amp Rosen B R (1995a) Visualmotion after effect in human cortical area MT revealed byfunctional magnetic resonance imaging Nature 11 139ndash141

Tootell R B H Reppas J B Kwong K K Malach R BornR T Brady T J Rosen B R amp Belliveau J W (1995b)Functional analysis of human MT and related-visual corticalareas using magnetic resonance imaging The Journal ofNeuroscience 15 3215 ndash3230

Treue S amp Maunsell J H R (1996) Attentional modulationof visual motion processing in cortical areas MT and MSTNature 382 539 ndash541

Van Essen D C Maunsell J H amp Bixby J L (1981) Themiddle temporal visual area in the macaque myeloarchi-tecture connections functional properties and topographicorganization Journal of Comparative Neurology 199 293 ndash326

54 Journal of Cognitive Neuroscience Volume 12 Number 1

since athletes were also presented the implied motioncondition) Half the subjects viewed these four differ-ent kinds of photographs passively To ensure atten-tion to stimuli from all conditions the other half ofthe subjects performed a lsquolsquo1-backrsquorsquo repetition detectiontask on the same sequences In a second experimentwe tested the response of area MTMST to photo-

graphs of animals and nature scenes that either de-picted implied motion or did not

RESULTS

The localizer scans (low contrast moving vs stationaryrings) successfully localized each subjectrsquos MTMST in

Figure 1 Functional data are overlaid on a high-resolution T1-weighted anatomical image for each slice Right hemisphere appears on the leftSignificance levels reflect the results of t-tests on the MR signal intensity ( plt10ndash7 equivalent to plt10ndash1 after Bonferroni correction) I Groupanalysis on functional data from 5 subjects (coregistered in Talairach space) showing regions responding significantly to (a) moving vs stationaryrings and (b) images with implied motion vs images without implied motion (Experiment 1) The green circles indicate regions activated significantlyfor both moving vs stationary rings and images with implied motion vs images without implied motion II Five slices from one subject showingactivation for viewing of (a) moving vs stationary rings and (b) images with implied motion vs images without implied motion (Experiment 1)

Kourtzi and Kanwisher 49

the lateral occipital region (Figure 1) consistent withprior reports (for example Tootell et al 1995b) Foreach subject this region served as the region of interest(ROI) from which the response was extracted for eachof the experimental conditions for the same subjectThe response for each condition and subject was quan-tified as the percent signal change (PSC) from thefixation baseline condition The average PSC acrosssubjects for each condition and the time course ofsignal intensity averaged across subjects are shown inFigure 2 for Experiment 1 and Figure 3 for Experiment2

For the first experiment a two-way ANOVA (StimulusTypepoundTask) on the PSC for each condition across sub-jects with Stimulus Type (implied motion athletes noimplied motion athletes people at rest houses) as thewithin-subjects variable and Task (passive 1-back) as thebetween-subjects variable showed a significant maineffect of Stimulus Type (F(3 18)=201 plt001) There

was no main effect of Task (F(1 18)lt1) and no inter-action of Stimulus Type and Task (F(3 18)=12 pgt3)The PSC in MTMST was significantly greater for imagesof athletes with implied motion vs athletes withoutimplied motion in both the passive (t(3)=35 plt05)and the 1-back (t(3)=45 plt05) tasks The PSC in MTMST during viewing of athletes without implied motionwas not significantly different from that for people atrest (t(7 )=06 pgt5)

The similar patterns of activation in MTMST acrosspassive viewing and 1-back tasks suggest that the ob-served activation is not likely to be due to differences intask difficulty or attentional allocation across conditionsIndeed the behavioral data from the 1-back task suggestthat this matching task was at least as difficult for imageswithout implied motion as for images with impliedmotion Specifically across three out of the four subjects(the behavioral data for one subject were lost due to acomputer error) the average percent correct detection

Figure 2 Results of Experiment 1 (a) An example stimulus from each condition Average percent signal increase (from the fixation baseline) andstandard deviations across subjects for each stimulus type in MT observed for each task (passive viewing 1-back) as well as the average across tasks(b) The time course of the percent change in MR signal intensity (from the fixation baseline) in MT over the period of the scan Black dot indicatesfixation IM images of athletes with implied motion no-IM images of athletes without implied motion R people at rest H houses

50 Journal of Cognitive Neuroscience Volume 12 Number 1

and total number of false alarms (in parentheses) overfour epochs for each condition were Implied motion90 (1) no implied motion 82 (1) people at rest93 (2) and Houses 74 (1)

The data from Experiment 2 were analyzed by a two-way repeated ANOVA (ConditionpoundStimulus Type) withCondition (implied motion vs no implied motion) andStimulus Type (animals vs nature scenes) as repeatedmeasures variables A main effect of Condition (F(13)=2685 plt001) was observed No main effect ofStimulus Type (F(1 3)=6847 p=079) nor a significantinteraction of Stimulus Type and Condition (F(1 3)lt1)was observed The PSC within MTMST was significantlygreater for implied than for no implied motion condi-tions for both animals (t(3)=37 plt05) and naturescenes (t(3)=61 plt01)

In order to look at regions of the brain beyond MTMST KolmogorovndashSmirnov statistics were run on eachvoxel scanned in each subject in Experiment 1 testingwhether that voxel showed stronger activation for (i)moving vs stationary rings in the localizer runs and (ii)implied motion athletes vs no implied motion athletes

For all subjects the lateral occipital regions thatshowed significant activation for moving vs stationaryrings in the localizer task overlapped with regionsshowing significant activation for implied motion vsno implied motion in the experimental runs Howeverfor all subjects the implied motion vs no impliedmotion athletes comparison also activated other re-gions contiguous to MTMST extending medially ante-riorly and posteriorly Six out of the eight subjects inExperiment 1 also showed significant activation forimplied motion vs no implied motion in the regionof the superior temporal sulcus Activations in theseregions were also observed in t-test group analyses offive subjects coregistered into Talairach space (Talair-ach amp Tournoux 1988) (Three subjects could not becoregistered due to poor resolution in the frontalregions of the brain as a result of surface coil usage)These analyses (see Figure 1) showed significantlystronger activations ( plt10ndash7 equivalent to plt10ndash1

after Bonferroni correction) for moving compared tostatic rings in MTMST and for implied motion vs noimplied motion

Figure 3 Results of Experiment 2 (a) An example stimulus from each condition Average percent signal change and standard deviations acrosssubjects for each stimulus type in MT ( b) The time course of the percent change in MR signal intensity in MT over the period of the scan Black dotindicates fixation AIM images of animals with implied motion Ano IM images of animals without implied motion SIM images of nature sceneswith implied motion Sno IM images of nature scenes without implied motion

Kourtzi and Kanwisher 51

DISCUSSION

Our results suggest that cortical areas involved in theanalysis of physical stimulus motion can be also en-gaged automatically by static images that merely implymotion Specifically passively observed static snapshotsof objects in action activate human motion areas (MTMST) more than static images of objects withoutimplied motion These results are observed for imagesimplying animate motion such as humans or animalsin action as well as inanimate motion such as activenature scenes

It is unlikely that these results can be explained bylow-level differences among the images in the differentconditions (for example differences in the location ofthe luminance edges) The activation in MTMST wassystematically greater for implied than no implied mo-tion across eight very different stimulus categories usedin the two experiments Furthermore it is unlikely thatthe modulation of activity in MT MST is related todifferences in image flicker (each photograph was dis-played for 300 msec followed by a 500-msec blankinterval followed by the next stimulus) since this flickeroccurred in all of our stimulus conditions

These results raise numerous questions about theanalysis of object motion in the human brain That isis MTMST involved in extracting implied motion infor-mation or is it influenced by such processes occurringelsewhere in the brain It seems unlikely that theperceptual analyses involved in the inference of motionfrom still images could be computed within MTMSTNeurophysiological and imaging studies have stronglysupported the role of MTMST in the analysis of stimulusmotion but not in processes such as object recognitionInferring motion from still images depends on objectcategorization and knowledge about the repertoire ofbehavior different objects can exhibit It seems mostlikely that such high-level perceptual inferences occurelsewhere in the brain and modulate activity in MTMSTin a top-down fashion Thus the observed activationsmay reflect an expectancy of object motion that could berepresented or influence representations in areas in-volved in processing physical stimulus motion (that isMTMST)

Consistent with this hypothesis the activation forimplied vs no implied motion extended beyond MTMST to several contiguous regions as shown in Figure 1These results are consistent with recent studies suggest-ing that other areas extending posterior and superior oranterior and inferior to MTMST are also involved inmotion analysis (De Jong Shipp Skidmore Frackowiakamp Zeki 1994 Dupont et al 1994 Shipp De Jong ZihlFrackowiak amp Zeki 1994 Watson et al 1993 ) Previousresearch has shown activation anterior and medial to MTfor passive viewing of images of illusory motion (ZekiWatson amp Frackowiak 1993) tool naming (MartinWiggs Ungerleider amp Haxby 1996) and the genera-

tion of action words (Martin Haxby Lalonde Wiggs ampUngerleider 1995) Recent imaging studies have shownactivation for motion boundaries in areas V3A (Tootellet al 1997 ) and KO (Orban Dupont De Bruyn VogelsVandenberghe amp Mortelmans 1995 Van OostendeSunaert Van Hecke Marchal amp Orban 1997 ) extendingposterior and medial to MT along the occipital surfaceThe activations observed in our subjects in the vicinity ofthe superior temporal sulcus are also consistent withprevious studies showing activation in the superiortemporal sulcus for motion imagery (Goebel et al1998) and viewing of biological motion stimuli (BondaPetrides Ostry amp Evans 1996 Puce Allison BentinGore amp McCarthy 1998)

Finally several prior findings support the hypothesisthat the current results reflect top-down influences ofhigh-level perceptual inferences on MTMST Both sin-gle unit (Treue amp Maunsell 1996 ) and fMRI studies(Beauchamp Cox amp DeYoe 1997 Corbetta MiezinDobmeyer Shulman amp Petersen 1990 1991 OrsquoCravenRosen Kwong Treisman amp Savoy 1997) have demon-strated that the response of MTMST to moving stimulican be strongly modulated by visual attention Alsoactivity in MTMST has been demonstrated even whensubjects close their eyes and merely imagine movingcompared to stationary arrays (Goebel et al 1998OrsquoCraven amp Kanwisher 1997 )

While the present work is consistent with these pre-vious studies suggesting that activation in MTMST canbe modulated in a top-down fashion we show here forthe first time that such top-down effects can occurautomatically That is dynamic information implicit inthe image was extracted and influenced activity in MTMST even though subjects were not asked or requiredto perceive attend to or imagine motion

One possible interpretation of our findings is thatinferring motion may involve or result in motion ima-gery Another interpretation is that the processing of aparticular object category (for example animals) maylead to activation of regions involved in processingproperties highly associated with that object category(for example motion) (Chao Haxby Lalonde Ungerlei-der amp Martin 1998 Martin et al 1996) Consistent withthe second hypothesis the current findings show thatactivation in MTMST is significantly higher for images ofpeople even people at rest than for images of houses

More broadly the current results support an emer-ging view of extrastriate cortex as playing a crucial rolenot only in visual perception but also in visual cogni-tion

METHODS

Subjects

Ten right-handed MIT students participated in Experi-ment 1 four in the passive viewing condition and six inthe 1-back matching condition Two subjects tested on

52 Journal of Cognitive Neuroscience Volume 12 Number 1

the 1-back matching condition were excluded from theanalysis due to excessive head motion Another six right-handed MIT students participated in Experiment 2 Twosubjects were excluded from the analysis in this condi-tion due to excessive head motion

Materials and Design

The stimuli used for functionally localizing MT were lowcontrast moving vs stationary concentric rings as de-scribed in Tootell et al (1995a) For the experimentalconditions stimuli were 300pound300 pixel digitized grays-cale photographs Experiment 1 involved a mixed de-sign with Stimulus Type a within-subject variable (withfour levels photographs of athletes with implied mo-tion athletes without implied motion people at restand houses) and Task a between-subjects factor (withtwo levels passive viewing vs 1-back repetition detec-tion) Experiment 2 involved two orthogonal factorscrossed within subjects Stimulus Type (animals vsscenes) and Condition (implied motion vs no impliedmotion)

Procedure

Each subject was run on two or more functional MTlocalizer scans with low contrast moving vs stationaryconcentric rings (as described in Tootell et al 1995a)Then each subject was run on four scans of the experi-mental test materials For the passive viewing condi-tions the subjects were asked to observe the imagescarefully while fixating a dot in the center of the image(Monitoring of eye movements outside the scanner forthree subjects that participated in Experiment 1 andthree subjects that participated in Experiment 2 showedthat the number of eye movements was very small in allconditions and did not differ significantly across condi-tions) For the 1-back matching condition subjects wereinstructed to press a button whenever they saw twoidentical pictures in a row Two or more repetitionsoccurred in each epoch

Each scan lasted 5 min and 36 sec and consisted ofsixteen 16-sec epochs with fixation periods interleavedas shown in Figures 2 and 3 Twenty different photo-graphs of the same type were presented in each epochEach photograph was presented for 300 msec with ablank interval of 500 msec between photographs Eachof the four stimulus types in each experiment werepresented in four different epochs within each scan ina design that balanced for the order of conditions asshown in Figures 2 and 3

MRI Acquisition

Scanning was done on the 3 T scanner (modified byANMR for Echo Planar Imaging) at the MGH-NMRCenter in Charlestown MA A custom bilateral surface

coil (built by J Thomas Vaughan) provided a high signal-to-noise ratio in posterior brain regions A bite-bar wasused to minimize head motion Standard imaging pro-cedures (Gradient Echo pulse sequence TR 2 sec TE30 msec flip angle 908 1808 offset 25 msec) were usedas described previously (Tong Nakayama Vaughan ampKanwisher 1998) Twelve 6-mm-thick near-coronal sliceswere oriented parallel to the brainstem and covered theoccipital lobe as well as the posterior portions of thetemporal and the parietal lobes One hundred sixty-eight functional images were collected for each slice ineach scan

Data Analysis

Each subjectrsquos MTMST was identified from the aver-age of the functional localizer scans as the set of allcontiguous voxels in the vicinity of the ascending limbof the inferior temporal sulcus (Tootell et al 1995bWatson et al 1993 Zeki et al 1991) that showedsignificantly stronger activation to moving comparedto static low-contrast concentric rings on a Kolmogor-ovndashSmirnov test at the level of plt0001 (uncorrected)In principle significant differences in KolmogorovndashSmirnov statistics can reflect differences in the var-iance only rather than in the means across conditions(Aguirre Zarahn amp DrsquoEsposito 1998) However thefact that the region selected by this procedure didindeed respond more strongly during the movingthan stationary conditions was confirmed by subse-quent analyses In particular t-tests across subjectsrevealed that the percent signal change in the selectedROIs was higher for moving than stationary conditions(a difference of 09 (t(7)=73 plt001) for Experi-ment 1 and 07 (t(3)=63 plt01) for Experiment 2)Moreover as shown in Figure 1 t-tests on the aver-aged group data for five subjects showed significantlystronger activation to moving compared to static rings( plt10ndash7 equivalent to plt10ndash1 after Bonferronicorrection)

For the analysis of the experimental scans the timecourse of MR signal intensity was extracted from eachsubjectrsquos MTMST by averaging the data from allvoxels within the ROI The average percent signalchange in MTMST was calculated for each subjectand stimulus type using the average signal intensityduring fixation epochs for the same subject experi-ment and task as a baseline Because the fMRIresponse typically lags four to six seconds after theneural response our data-analysis procedure treatedthe first image in each epoch as belonging to thecondition of the preceding epoch and omitted thenext two images (during the transition betweenepochs) from the analysis

An ANOVA across subjects was run on the averagepercent signal change in each of the conditions in eachexperiment Because data were analysed within inde-

Kourtzi and Kanwisher 53

pendently defined ROIs for MTMST no correction formultiple voxelwise comparisons was required

Acknowledgments

We would like to thank Ted Adelson and Maggie Shiffrar fortheir helpful comments and suggestions on this project andBruce Rosen and many people at the MGH-NMR Center fortechnical assistance and support We would also like to thankPaul Downing and Russell Epstein for their comments onprevious versions of this manuscript This research wassupported by NIMH Grant 56037 and a Human Frontiersgrant to Nancy Kanwisher Some of the results discussed in thismanuscript were first presented at the 1998 meeting of theSociety for Neuroscience in Los Angeles CA and the 1998meeting of the Psychonomic Society in Dallas TX

Reprint requests should be sent to Zoe Kourtzi Dept of Brainand Cognitive Science MIT NE20-4043 77 Massachusetts AveCambridge MA 02139-4307 or via e-mail zoepsychemitedu

REFERENCES

Aguirre G K Zarahn E amp DrsquoEsposito M (1998) A critique ofthe use of the KolmogorovndashSmirnov (KS) statistic for theanalysis of BOLD fMRI data Magnetic Resonance in Medi-cine 39 500 ndash505

Beauchamp M S Cox R W amp DeYoe E A (1997) Gradedeffects of spatial and featural attention on human-area MTand associated motion-processing areas Journal of Neuro-physiology 78 516 ndash520

Bonda E Petrides M Ostry D amp Evans A (1996) Specificinvolvement of human-parietal systems and the amygdalain the perception of biological motion The Journal ofNeuroscience 16 3737ndash 3744

Britten K H Newsome W T Shalden M N Celebrini S ampMovshon J A (1996) A relationship between behavioralchoice and the visual responses of neurons in macaque MTVisual Neuroscience 13 87ndash100

Chao L L Haxby J V Lalonde F M Ungerleider L G ampMartin A (1998) Pictures of animals and tools deferen-tially engage object form-related and motion-related brainregions Paper presented at the 28th Annual Meeting of theSociety for Neuroscience LA CA

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1990) Attentional modulation of neuralprocessing of shape color and velocity in humans Science248 1556 ndash1559

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1991) Selective and divided attention duringvisual discriminations of shape color and speed Functionalanatomy by positron emission tomography Journal ofNeuroscience 11 2383ndash2402

De Jong B M Shipp S Skidmore B Frackowiak R S Jamp Zeki S (1994) The cerebral activity related to visualperception of forward motion in depth Brain 117 1039ndash1054

Dubner R amp Zeki S M (1971) Response properties andreceptive fields of cells in an anatomically-defined region ofthe superior temporal sulcus in the monkey Brain Re-search 35 528ndash532

Dupont P Orban G A De Bruyn B Verbruggen A ampMortelmans L (1994) Many areas in the human brain re-spond to visual motion Journal of Neurophysiology 721420ndash1424

Freyd J (1983) The mental representation of movement

when static stimuli are viewed Perception and Psycho-physics 33 575ndash581

Goebel R Khorram-Sefat D Muckli L Hacker H amp SingerW (1998) The constructive nature of vision Direct evidencefrom functional magnetic resonance imaging studies of ap-parent motion and motion imagery European Journal ofNeuroscience 10 1563 ndash1573

Kaneoke Y Bundou M Koyama S Suzuki H amp Kakigi R(1997) Human cortical area responding to stimuli in ap-parent motion NeuroReport 8 677ndash 682

Martin A Haxby J V Lalonde F M Wiggs C L amp Unger-leider L G (1995) Discrete cortical regions associated withknowledge of color and knowledge of action Science 270102 ndash105

Martin A Wiggs C L Ungerleider L G amp Haxby J V(1996) Neural correlates of category-specific knowledgeNature 379 649 ndash652

Maunsell J H amp Van Essen D C (1983) The connections ofthe middle temporal-visual area (MT) and their relationshipto a cortical hierarchy in the macaque monkey Journal ofNeuroscience 3 2563 ndash2586

OrsquoCraven K M amp Kanwisher N G (1997) Visual imagery ofmoving stimuli activates area MTMST Paper presented atthe 27th Annual Meeting of the Society for NeuroscienceNew Orleans LA

OrsquoCraven K M Rosen B R Kwong K K Treisman A ampSavoy R L (1997) Voluntary attention modulates fMRI ac-tivity in human MT-MST Neuron 18 591ndash 598

Orban G A Dupont P De Bruyn B Vogels R Vanden-berghe R amp Mortelmans L (1995) A motion area in humanvisual cortex Proceedings of the National Academy ofScience 92 993 ndash997

Puce A Allison T Bentin S Gore J C amp McCarthy G(1998) Temporal-cortex activation in human subjects view-ing eye and mouth movements Journal of Neuroscience18 2188 ndash2199

Shipp S de Jong B M Zihl J Frackowiak R S J amp ZekiS (1994) The brain activity related to residual motion vi-sion in a patient with bilateral lesions of V5 Brain 1171023 ndash1038

Talairach J amp Tournoux P (1988) Co-planar stereotaxicatlas of the human brain New York Thieme Medical

Tong F Nakayama K Vaughan J T amp Kanwisher N (1998)Binocular rivalry and visual awareness in human extrastriatecortex Neuron 21 753 ndash759

Tootell R B H Mendola J D Hadjikhani N K Ledden PJ Liu A K Reppas J B Sereno M I amp Dale A M(1997) Functional analysis of V3A and related areas in hu-man visual cortex The Journal of Neuroscience 15 7060 ndash7078

Tootell R B H Reppas J B Dale A M Look R B SerenoM I Malach R Brady T J amp Rosen B R (1995a) Visualmotion after effect in human cortical area MT revealed byfunctional magnetic resonance imaging Nature 11 139ndash141

Tootell R B H Reppas J B Kwong K K Malach R BornR T Brady T J Rosen B R amp Belliveau J W (1995b)Functional analysis of human MT and related-visual corticalareas using magnetic resonance imaging The Journal ofNeuroscience 15 3215 ndash3230

Treue S amp Maunsell J H R (1996) Attentional modulationof visual motion processing in cortical areas MT and MSTNature 382 539 ndash541

Van Essen D C Maunsell J H amp Bixby J L (1981) Themiddle temporal visual area in the macaque myeloarchi-tecture connections functional properties and topographicorganization Journal of Comparative Neurology 199 293 ndash326

54 Journal of Cognitive Neuroscience Volume 12 Number 1

the lateral occipital region (Figure 1) consistent withprior reports (for example Tootell et al 1995b) Foreach subject this region served as the region of interest(ROI) from which the response was extracted for eachof the experimental conditions for the same subjectThe response for each condition and subject was quan-tified as the percent signal change (PSC) from thefixation baseline condition The average PSC acrosssubjects for each condition and the time course ofsignal intensity averaged across subjects are shown inFigure 2 for Experiment 1 and Figure 3 for Experiment2

For the first experiment a two-way ANOVA (StimulusTypepoundTask) on the PSC for each condition across sub-jects with Stimulus Type (implied motion athletes noimplied motion athletes people at rest houses) as thewithin-subjects variable and Task (passive 1-back) as thebetween-subjects variable showed a significant maineffect of Stimulus Type (F(3 18)=201 plt001) There

was no main effect of Task (F(1 18)lt1) and no inter-action of Stimulus Type and Task (F(3 18)=12 pgt3)The PSC in MTMST was significantly greater for imagesof athletes with implied motion vs athletes withoutimplied motion in both the passive (t(3)=35 plt05)and the 1-back (t(3)=45 plt05) tasks The PSC in MTMST during viewing of athletes without implied motionwas not significantly different from that for people atrest (t(7 )=06 pgt5)

The similar patterns of activation in MTMST acrosspassive viewing and 1-back tasks suggest that the ob-served activation is not likely to be due to differences intask difficulty or attentional allocation across conditionsIndeed the behavioral data from the 1-back task suggestthat this matching task was at least as difficult for imageswithout implied motion as for images with impliedmotion Specifically across three out of the four subjects(the behavioral data for one subject were lost due to acomputer error) the average percent correct detection

Figure 2 Results of Experiment 1 (a) An example stimulus from each condition Average percent signal increase (from the fixation baseline) andstandard deviations across subjects for each stimulus type in MT observed for each task (passive viewing 1-back) as well as the average across tasks(b) The time course of the percent change in MR signal intensity (from the fixation baseline) in MT over the period of the scan Black dot indicatesfixation IM images of athletes with implied motion no-IM images of athletes without implied motion R people at rest H houses

50 Journal of Cognitive Neuroscience Volume 12 Number 1

and total number of false alarms (in parentheses) overfour epochs for each condition were Implied motion90 (1) no implied motion 82 (1) people at rest93 (2) and Houses 74 (1)

The data from Experiment 2 were analyzed by a two-way repeated ANOVA (ConditionpoundStimulus Type) withCondition (implied motion vs no implied motion) andStimulus Type (animals vs nature scenes) as repeatedmeasures variables A main effect of Condition (F(13)=2685 plt001) was observed No main effect ofStimulus Type (F(1 3)=6847 p=079) nor a significantinteraction of Stimulus Type and Condition (F(1 3)lt1)was observed The PSC within MTMST was significantlygreater for implied than for no implied motion condi-tions for both animals (t(3)=37 plt05) and naturescenes (t(3)=61 plt01)

In order to look at regions of the brain beyond MTMST KolmogorovndashSmirnov statistics were run on eachvoxel scanned in each subject in Experiment 1 testingwhether that voxel showed stronger activation for (i)moving vs stationary rings in the localizer runs and (ii)implied motion athletes vs no implied motion athletes

For all subjects the lateral occipital regions thatshowed significant activation for moving vs stationaryrings in the localizer task overlapped with regionsshowing significant activation for implied motion vsno implied motion in the experimental runs Howeverfor all subjects the implied motion vs no impliedmotion athletes comparison also activated other re-gions contiguous to MTMST extending medially ante-riorly and posteriorly Six out of the eight subjects inExperiment 1 also showed significant activation forimplied motion vs no implied motion in the regionof the superior temporal sulcus Activations in theseregions were also observed in t-test group analyses offive subjects coregistered into Talairach space (Talair-ach amp Tournoux 1988) (Three subjects could not becoregistered due to poor resolution in the frontalregions of the brain as a result of surface coil usage)These analyses (see Figure 1) showed significantlystronger activations ( plt10ndash7 equivalent to plt10ndash1

after Bonferroni correction) for moving compared tostatic rings in MTMST and for implied motion vs noimplied motion

Figure 3 Results of Experiment 2 (a) An example stimulus from each condition Average percent signal change and standard deviations acrosssubjects for each stimulus type in MT ( b) The time course of the percent change in MR signal intensity in MT over the period of the scan Black dotindicates fixation AIM images of animals with implied motion Ano IM images of animals without implied motion SIM images of nature sceneswith implied motion Sno IM images of nature scenes without implied motion

Kourtzi and Kanwisher 51

DISCUSSION

Our results suggest that cortical areas involved in theanalysis of physical stimulus motion can be also en-gaged automatically by static images that merely implymotion Specifically passively observed static snapshotsof objects in action activate human motion areas (MTMST) more than static images of objects withoutimplied motion These results are observed for imagesimplying animate motion such as humans or animalsin action as well as inanimate motion such as activenature scenes

It is unlikely that these results can be explained bylow-level differences among the images in the differentconditions (for example differences in the location ofthe luminance edges) The activation in MTMST wassystematically greater for implied than no implied mo-tion across eight very different stimulus categories usedin the two experiments Furthermore it is unlikely thatthe modulation of activity in MT MST is related todifferences in image flicker (each photograph was dis-played for 300 msec followed by a 500-msec blankinterval followed by the next stimulus) since this flickeroccurred in all of our stimulus conditions

These results raise numerous questions about theanalysis of object motion in the human brain That isis MTMST involved in extracting implied motion infor-mation or is it influenced by such processes occurringelsewhere in the brain It seems unlikely that theperceptual analyses involved in the inference of motionfrom still images could be computed within MTMSTNeurophysiological and imaging studies have stronglysupported the role of MTMST in the analysis of stimulusmotion but not in processes such as object recognitionInferring motion from still images depends on objectcategorization and knowledge about the repertoire ofbehavior different objects can exhibit It seems mostlikely that such high-level perceptual inferences occurelsewhere in the brain and modulate activity in MTMSTin a top-down fashion Thus the observed activationsmay reflect an expectancy of object motion that could berepresented or influence representations in areas in-volved in processing physical stimulus motion (that isMTMST)

Consistent with this hypothesis the activation forimplied vs no implied motion extended beyond MTMST to several contiguous regions as shown in Figure 1These results are consistent with recent studies suggest-ing that other areas extending posterior and superior oranterior and inferior to MTMST are also involved inmotion analysis (De Jong Shipp Skidmore Frackowiakamp Zeki 1994 Dupont et al 1994 Shipp De Jong ZihlFrackowiak amp Zeki 1994 Watson et al 1993 ) Previousresearch has shown activation anterior and medial to MTfor passive viewing of images of illusory motion (ZekiWatson amp Frackowiak 1993) tool naming (MartinWiggs Ungerleider amp Haxby 1996) and the genera-

tion of action words (Martin Haxby Lalonde Wiggs ampUngerleider 1995) Recent imaging studies have shownactivation for motion boundaries in areas V3A (Tootellet al 1997 ) and KO (Orban Dupont De Bruyn VogelsVandenberghe amp Mortelmans 1995 Van OostendeSunaert Van Hecke Marchal amp Orban 1997 ) extendingposterior and medial to MT along the occipital surfaceThe activations observed in our subjects in the vicinity ofthe superior temporal sulcus are also consistent withprevious studies showing activation in the superiortemporal sulcus for motion imagery (Goebel et al1998) and viewing of biological motion stimuli (BondaPetrides Ostry amp Evans 1996 Puce Allison BentinGore amp McCarthy 1998)

Finally several prior findings support the hypothesisthat the current results reflect top-down influences ofhigh-level perceptual inferences on MTMST Both sin-gle unit (Treue amp Maunsell 1996 ) and fMRI studies(Beauchamp Cox amp DeYoe 1997 Corbetta MiezinDobmeyer Shulman amp Petersen 1990 1991 OrsquoCravenRosen Kwong Treisman amp Savoy 1997) have demon-strated that the response of MTMST to moving stimulican be strongly modulated by visual attention Alsoactivity in MTMST has been demonstrated even whensubjects close their eyes and merely imagine movingcompared to stationary arrays (Goebel et al 1998OrsquoCraven amp Kanwisher 1997 )

While the present work is consistent with these pre-vious studies suggesting that activation in MTMST canbe modulated in a top-down fashion we show here forthe first time that such top-down effects can occurautomatically That is dynamic information implicit inthe image was extracted and influenced activity in MTMST even though subjects were not asked or requiredto perceive attend to or imagine motion

One possible interpretation of our findings is thatinferring motion may involve or result in motion ima-gery Another interpretation is that the processing of aparticular object category (for example animals) maylead to activation of regions involved in processingproperties highly associated with that object category(for example motion) (Chao Haxby Lalonde Ungerlei-der amp Martin 1998 Martin et al 1996) Consistent withthe second hypothesis the current findings show thatactivation in MTMST is significantly higher for images ofpeople even people at rest than for images of houses

More broadly the current results support an emer-ging view of extrastriate cortex as playing a crucial rolenot only in visual perception but also in visual cogni-tion

METHODS

Subjects

Ten right-handed MIT students participated in Experi-ment 1 four in the passive viewing condition and six inthe 1-back matching condition Two subjects tested on

52 Journal of Cognitive Neuroscience Volume 12 Number 1

the 1-back matching condition were excluded from theanalysis due to excessive head motion Another six right-handed MIT students participated in Experiment 2 Twosubjects were excluded from the analysis in this condi-tion due to excessive head motion

Materials and Design

The stimuli used for functionally localizing MT were lowcontrast moving vs stationary concentric rings as de-scribed in Tootell et al (1995a) For the experimentalconditions stimuli were 300pound300 pixel digitized grays-cale photographs Experiment 1 involved a mixed de-sign with Stimulus Type a within-subject variable (withfour levels photographs of athletes with implied mo-tion athletes without implied motion people at restand houses) and Task a between-subjects factor (withtwo levels passive viewing vs 1-back repetition detec-tion) Experiment 2 involved two orthogonal factorscrossed within subjects Stimulus Type (animals vsscenes) and Condition (implied motion vs no impliedmotion)

Procedure

Each subject was run on two or more functional MTlocalizer scans with low contrast moving vs stationaryconcentric rings (as described in Tootell et al 1995a)Then each subject was run on four scans of the experi-mental test materials For the passive viewing condi-tions the subjects were asked to observe the imagescarefully while fixating a dot in the center of the image(Monitoring of eye movements outside the scanner forthree subjects that participated in Experiment 1 andthree subjects that participated in Experiment 2 showedthat the number of eye movements was very small in allconditions and did not differ significantly across condi-tions) For the 1-back matching condition subjects wereinstructed to press a button whenever they saw twoidentical pictures in a row Two or more repetitionsoccurred in each epoch

Each scan lasted 5 min and 36 sec and consisted ofsixteen 16-sec epochs with fixation periods interleavedas shown in Figures 2 and 3 Twenty different photo-graphs of the same type were presented in each epochEach photograph was presented for 300 msec with ablank interval of 500 msec between photographs Eachof the four stimulus types in each experiment werepresented in four different epochs within each scan ina design that balanced for the order of conditions asshown in Figures 2 and 3

MRI Acquisition

Scanning was done on the 3 T scanner (modified byANMR for Echo Planar Imaging) at the MGH-NMRCenter in Charlestown MA A custom bilateral surface

coil (built by J Thomas Vaughan) provided a high signal-to-noise ratio in posterior brain regions A bite-bar wasused to minimize head motion Standard imaging pro-cedures (Gradient Echo pulse sequence TR 2 sec TE30 msec flip angle 908 1808 offset 25 msec) were usedas described previously (Tong Nakayama Vaughan ampKanwisher 1998) Twelve 6-mm-thick near-coronal sliceswere oriented parallel to the brainstem and covered theoccipital lobe as well as the posterior portions of thetemporal and the parietal lobes One hundred sixty-eight functional images were collected for each slice ineach scan

Data Analysis

Each subjectrsquos MTMST was identified from the aver-age of the functional localizer scans as the set of allcontiguous voxels in the vicinity of the ascending limbof the inferior temporal sulcus (Tootell et al 1995bWatson et al 1993 Zeki et al 1991) that showedsignificantly stronger activation to moving comparedto static low-contrast concentric rings on a Kolmogor-ovndashSmirnov test at the level of plt0001 (uncorrected)In principle significant differences in KolmogorovndashSmirnov statistics can reflect differences in the var-iance only rather than in the means across conditions(Aguirre Zarahn amp DrsquoEsposito 1998) However thefact that the region selected by this procedure didindeed respond more strongly during the movingthan stationary conditions was confirmed by subse-quent analyses In particular t-tests across subjectsrevealed that the percent signal change in the selectedROIs was higher for moving than stationary conditions(a difference of 09 (t(7)=73 plt001) for Experi-ment 1 and 07 (t(3)=63 plt01) for Experiment 2)Moreover as shown in Figure 1 t-tests on the aver-aged group data for five subjects showed significantlystronger activation to moving compared to static rings( plt10ndash7 equivalent to plt10ndash1 after Bonferronicorrection)

For the analysis of the experimental scans the timecourse of MR signal intensity was extracted from eachsubjectrsquos MTMST by averaging the data from allvoxels within the ROI The average percent signalchange in MTMST was calculated for each subjectand stimulus type using the average signal intensityduring fixation epochs for the same subject experi-ment and task as a baseline Because the fMRIresponse typically lags four to six seconds after theneural response our data-analysis procedure treatedthe first image in each epoch as belonging to thecondition of the preceding epoch and omitted thenext two images (during the transition betweenepochs) from the analysis

An ANOVA across subjects was run on the averagepercent signal change in each of the conditions in eachexperiment Because data were analysed within inde-

Kourtzi and Kanwisher 53

pendently defined ROIs for MTMST no correction formultiple voxelwise comparisons was required

Acknowledgments

We would like to thank Ted Adelson and Maggie Shiffrar fortheir helpful comments and suggestions on this project andBruce Rosen and many people at the MGH-NMR Center fortechnical assistance and support We would also like to thankPaul Downing and Russell Epstein for their comments onprevious versions of this manuscript This research wassupported by NIMH Grant 56037 and a Human Frontiersgrant to Nancy Kanwisher Some of the results discussed in thismanuscript were first presented at the 1998 meeting of theSociety for Neuroscience in Los Angeles CA and the 1998meeting of the Psychonomic Society in Dallas TX

Reprint requests should be sent to Zoe Kourtzi Dept of Brainand Cognitive Science MIT NE20-4043 77 Massachusetts AveCambridge MA 02139-4307 or via e-mail zoepsychemitedu

REFERENCES

Aguirre G K Zarahn E amp DrsquoEsposito M (1998) A critique ofthe use of the KolmogorovndashSmirnov (KS) statistic for theanalysis of BOLD fMRI data Magnetic Resonance in Medi-cine 39 500 ndash505

Beauchamp M S Cox R W amp DeYoe E A (1997) Gradedeffects of spatial and featural attention on human-area MTand associated motion-processing areas Journal of Neuro-physiology 78 516 ndash520

Bonda E Petrides M Ostry D amp Evans A (1996) Specificinvolvement of human-parietal systems and the amygdalain the perception of biological motion The Journal ofNeuroscience 16 3737ndash 3744

Britten K H Newsome W T Shalden M N Celebrini S ampMovshon J A (1996) A relationship between behavioralchoice and the visual responses of neurons in macaque MTVisual Neuroscience 13 87ndash100

Chao L L Haxby J V Lalonde F M Ungerleider L G ampMartin A (1998) Pictures of animals and tools deferen-tially engage object form-related and motion-related brainregions Paper presented at the 28th Annual Meeting of theSociety for Neuroscience LA CA

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1990) Attentional modulation of neuralprocessing of shape color and velocity in humans Science248 1556 ndash1559

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1991) Selective and divided attention duringvisual discriminations of shape color and speed Functionalanatomy by positron emission tomography Journal ofNeuroscience 11 2383ndash2402

De Jong B M Shipp S Skidmore B Frackowiak R S Jamp Zeki S (1994) The cerebral activity related to visualperception of forward motion in depth Brain 117 1039ndash1054

Dubner R amp Zeki S M (1971) Response properties andreceptive fields of cells in an anatomically-defined region ofthe superior temporal sulcus in the monkey Brain Re-search 35 528ndash532

Dupont P Orban G A De Bruyn B Verbruggen A ampMortelmans L (1994) Many areas in the human brain re-spond to visual motion Journal of Neurophysiology 721420ndash1424

Freyd J (1983) The mental representation of movement

when static stimuli are viewed Perception and Psycho-physics 33 575ndash581

Goebel R Khorram-Sefat D Muckli L Hacker H amp SingerW (1998) The constructive nature of vision Direct evidencefrom functional magnetic resonance imaging studies of ap-parent motion and motion imagery European Journal ofNeuroscience 10 1563 ndash1573

Kaneoke Y Bundou M Koyama S Suzuki H amp Kakigi R(1997) Human cortical area responding to stimuli in ap-parent motion NeuroReport 8 677ndash 682

Martin A Haxby J V Lalonde F M Wiggs C L amp Unger-leider L G (1995) Discrete cortical regions associated withknowledge of color and knowledge of action Science 270102 ndash105

Martin A Wiggs C L Ungerleider L G amp Haxby J V(1996) Neural correlates of category-specific knowledgeNature 379 649 ndash652

Maunsell J H amp Van Essen D C (1983) The connections ofthe middle temporal-visual area (MT) and their relationshipto a cortical hierarchy in the macaque monkey Journal ofNeuroscience 3 2563 ndash2586

OrsquoCraven K M amp Kanwisher N G (1997) Visual imagery ofmoving stimuli activates area MTMST Paper presented atthe 27th Annual Meeting of the Society for NeuroscienceNew Orleans LA

OrsquoCraven K M Rosen B R Kwong K K Treisman A ampSavoy R L (1997) Voluntary attention modulates fMRI ac-tivity in human MT-MST Neuron 18 591ndash 598

Orban G A Dupont P De Bruyn B Vogels R Vanden-berghe R amp Mortelmans L (1995) A motion area in humanvisual cortex Proceedings of the National Academy ofScience 92 993 ndash997

Puce A Allison T Bentin S Gore J C amp McCarthy G(1998) Temporal-cortex activation in human subjects view-ing eye and mouth movements Journal of Neuroscience18 2188 ndash2199

Shipp S de Jong B M Zihl J Frackowiak R S J amp ZekiS (1994) The brain activity related to residual motion vi-sion in a patient with bilateral lesions of V5 Brain 1171023 ndash1038

Talairach J amp Tournoux P (1988) Co-planar stereotaxicatlas of the human brain New York Thieme Medical

Tong F Nakayama K Vaughan J T amp Kanwisher N (1998)Binocular rivalry and visual awareness in human extrastriatecortex Neuron 21 753 ndash759

Tootell R B H Mendola J D Hadjikhani N K Ledden PJ Liu A K Reppas J B Sereno M I amp Dale A M(1997) Functional analysis of V3A and related areas in hu-man visual cortex The Journal of Neuroscience 15 7060 ndash7078

Tootell R B H Reppas J B Dale A M Look R B SerenoM I Malach R Brady T J amp Rosen B R (1995a) Visualmotion after effect in human cortical area MT revealed byfunctional magnetic resonance imaging Nature 11 139ndash141

Tootell R B H Reppas J B Kwong K K Malach R BornR T Brady T J Rosen B R amp Belliveau J W (1995b)Functional analysis of human MT and related-visual corticalareas using magnetic resonance imaging The Journal ofNeuroscience 15 3215 ndash3230

Treue S amp Maunsell J H R (1996) Attentional modulationof visual motion processing in cortical areas MT and MSTNature 382 539 ndash541

Van Essen D C Maunsell J H amp Bixby J L (1981) Themiddle temporal visual area in the macaque myeloarchi-tecture connections functional properties and topographicorganization Journal of Comparative Neurology 199 293 ndash326

54 Journal of Cognitive Neuroscience Volume 12 Number 1

and total number of false alarms (in parentheses) overfour epochs for each condition were Implied motion90 (1) no implied motion 82 (1) people at rest93 (2) and Houses 74 (1)

The data from Experiment 2 were analyzed by a two-way repeated ANOVA (ConditionpoundStimulus Type) withCondition (implied motion vs no implied motion) andStimulus Type (animals vs nature scenes) as repeatedmeasures variables A main effect of Condition (F(13)=2685 plt001) was observed No main effect ofStimulus Type (F(1 3)=6847 p=079) nor a significantinteraction of Stimulus Type and Condition (F(1 3)lt1)was observed The PSC within MTMST was significantlygreater for implied than for no implied motion condi-tions for both animals (t(3)=37 plt05) and naturescenes (t(3)=61 plt01)

In order to look at regions of the brain beyond MTMST KolmogorovndashSmirnov statistics were run on eachvoxel scanned in each subject in Experiment 1 testingwhether that voxel showed stronger activation for (i)moving vs stationary rings in the localizer runs and (ii)implied motion athletes vs no implied motion athletes

For all subjects the lateral occipital regions thatshowed significant activation for moving vs stationaryrings in the localizer task overlapped with regionsshowing significant activation for implied motion vsno implied motion in the experimental runs Howeverfor all subjects the implied motion vs no impliedmotion athletes comparison also activated other re-gions contiguous to MTMST extending medially ante-riorly and posteriorly Six out of the eight subjects inExperiment 1 also showed significant activation forimplied motion vs no implied motion in the regionof the superior temporal sulcus Activations in theseregions were also observed in t-test group analyses offive subjects coregistered into Talairach space (Talair-ach amp Tournoux 1988) (Three subjects could not becoregistered due to poor resolution in the frontalregions of the brain as a result of surface coil usage)These analyses (see Figure 1) showed significantlystronger activations ( plt10ndash7 equivalent to plt10ndash1

after Bonferroni correction) for moving compared tostatic rings in MTMST and for implied motion vs noimplied motion

Figure 3 Results of Experiment 2 (a) An example stimulus from each condition Average percent signal change and standard deviations acrosssubjects for each stimulus type in MT ( b) The time course of the percent change in MR signal intensity in MT over the period of the scan Black dotindicates fixation AIM images of animals with implied motion Ano IM images of animals without implied motion SIM images of nature sceneswith implied motion Sno IM images of nature scenes without implied motion

Kourtzi and Kanwisher 51

DISCUSSION

Our results suggest that cortical areas involved in theanalysis of physical stimulus motion can be also en-gaged automatically by static images that merely implymotion Specifically passively observed static snapshotsof objects in action activate human motion areas (MTMST) more than static images of objects withoutimplied motion These results are observed for imagesimplying animate motion such as humans or animalsin action as well as inanimate motion such as activenature scenes

It is unlikely that these results can be explained bylow-level differences among the images in the differentconditions (for example differences in the location ofthe luminance edges) The activation in MTMST wassystematically greater for implied than no implied mo-tion across eight very different stimulus categories usedin the two experiments Furthermore it is unlikely thatthe modulation of activity in MT MST is related todifferences in image flicker (each photograph was dis-played for 300 msec followed by a 500-msec blankinterval followed by the next stimulus) since this flickeroccurred in all of our stimulus conditions

These results raise numerous questions about theanalysis of object motion in the human brain That isis MTMST involved in extracting implied motion infor-mation or is it influenced by such processes occurringelsewhere in the brain It seems unlikely that theperceptual analyses involved in the inference of motionfrom still images could be computed within MTMSTNeurophysiological and imaging studies have stronglysupported the role of MTMST in the analysis of stimulusmotion but not in processes such as object recognitionInferring motion from still images depends on objectcategorization and knowledge about the repertoire ofbehavior different objects can exhibit It seems mostlikely that such high-level perceptual inferences occurelsewhere in the brain and modulate activity in MTMSTin a top-down fashion Thus the observed activationsmay reflect an expectancy of object motion that could berepresented or influence representations in areas in-volved in processing physical stimulus motion (that isMTMST)

Consistent with this hypothesis the activation forimplied vs no implied motion extended beyond MTMST to several contiguous regions as shown in Figure 1These results are consistent with recent studies suggest-ing that other areas extending posterior and superior oranterior and inferior to MTMST are also involved inmotion analysis (De Jong Shipp Skidmore Frackowiakamp Zeki 1994 Dupont et al 1994 Shipp De Jong ZihlFrackowiak amp Zeki 1994 Watson et al 1993 ) Previousresearch has shown activation anterior and medial to MTfor passive viewing of images of illusory motion (ZekiWatson amp Frackowiak 1993) tool naming (MartinWiggs Ungerleider amp Haxby 1996) and the genera-

tion of action words (Martin Haxby Lalonde Wiggs ampUngerleider 1995) Recent imaging studies have shownactivation for motion boundaries in areas V3A (Tootellet al 1997 ) and KO (Orban Dupont De Bruyn VogelsVandenberghe amp Mortelmans 1995 Van OostendeSunaert Van Hecke Marchal amp Orban 1997 ) extendingposterior and medial to MT along the occipital surfaceThe activations observed in our subjects in the vicinity ofthe superior temporal sulcus are also consistent withprevious studies showing activation in the superiortemporal sulcus for motion imagery (Goebel et al1998) and viewing of biological motion stimuli (BondaPetrides Ostry amp Evans 1996 Puce Allison BentinGore amp McCarthy 1998)

Finally several prior findings support the hypothesisthat the current results reflect top-down influences ofhigh-level perceptual inferences on MTMST Both sin-gle unit (Treue amp Maunsell 1996 ) and fMRI studies(Beauchamp Cox amp DeYoe 1997 Corbetta MiezinDobmeyer Shulman amp Petersen 1990 1991 OrsquoCravenRosen Kwong Treisman amp Savoy 1997) have demon-strated that the response of MTMST to moving stimulican be strongly modulated by visual attention Alsoactivity in MTMST has been demonstrated even whensubjects close their eyes and merely imagine movingcompared to stationary arrays (Goebel et al 1998OrsquoCraven amp Kanwisher 1997 )

While the present work is consistent with these pre-vious studies suggesting that activation in MTMST canbe modulated in a top-down fashion we show here forthe first time that such top-down effects can occurautomatically That is dynamic information implicit inthe image was extracted and influenced activity in MTMST even though subjects were not asked or requiredto perceive attend to or imagine motion

One possible interpretation of our findings is thatinferring motion may involve or result in motion ima-gery Another interpretation is that the processing of aparticular object category (for example animals) maylead to activation of regions involved in processingproperties highly associated with that object category(for example motion) (Chao Haxby Lalonde Ungerlei-der amp Martin 1998 Martin et al 1996) Consistent withthe second hypothesis the current findings show thatactivation in MTMST is significantly higher for images ofpeople even people at rest than for images of houses

More broadly the current results support an emer-ging view of extrastriate cortex as playing a crucial rolenot only in visual perception but also in visual cogni-tion

METHODS

Subjects

Ten right-handed MIT students participated in Experi-ment 1 four in the passive viewing condition and six inthe 1-back matching condition Two subjects tested on

52 Journal of Cognitive Neuroscience Volume 12 Number 1

the 1-back matching condition were excluded from theanalysis due to excessive head motion Another six right-handed MIT students participated in Experiment 2 Twosubjects were excluded from the analysis in this condi-tion due to excessive head motion

Materials and Design

The stimuli used for functionally localizing MT were lowcontrast moving vs stationary concentric rings as de-scribed in Tootell et al (1995a) For the experimentalconditions stimuli were 300pound300 pixel digitized grays-cale photographs Experiment 1 involved a mixed de-sign with Stimulus Type a within-subject variable (withfour levels photographs of athletes with implied mo-tion athletes without implied motion people at restand houses) and Task a between-subjects factor (withtwo levels passive viewing vs 1-back repetition detec-tion) Experiment 2 involved two orthogonal factorscrossed within subjects Stimulus Type (animals vsscenes) and Condition (implied motion vs no impliedmotion)

Procedure

Each subject was run on two or more functional MTlocalizer scans with low contrast moving vs stationaryconcentric rings (as described in Tootell et al 1995a)Then each subject was run on four scans of the experi-mental test materials For the passive viewing condi-tions the subjects were asked to observe the imagescarefully while fixating a dot in the center of the image(Monitoring of eye movements outside the scanner forthree subjects that participated in Experiment 1 andthree subjects that participated in Experiment 2 showedthat the number of eye movements was very small in allconditions and did not differ significantly across condi-tions) For the 1-back matching condition subjects wereinstructed to press a button whenever they saw twoidentical pictures in a row Two or more repetitionsoccurred in each epoch

Each scan lasted 5 min and 36 sec and consisted ofsixteen 16-sec epochs with fixation periods interleavedas shown in Figures 2 and 3 Twenty different photo-graphs of the same type were presented in each epochEach photograph was presented for 300 msec with ablank interval of 500 msec between photographs Eachof the four stimulus types in each experiment werepresented in four different epochs within each scan ina design that balanced for the order of conditions asshown in Figures 2 and 3

MRI Acquisition

Scanning was done on the 3 T scanner (modified byANMR for Echo Planar Imaging) at the MGH-NMRCenter in Charlestown MA A custom bilateral surface

coil (built by J Thomas Vaughan) provided a high signal-to-noise ratio in posterior brain regions A bite-bar wasused to minimize head motion Standard imaging pro-cedures (Gradient Echo pulse sequence TR 2 sec TE30 msec flip angle 908 1808 offset 25 msec) were usedas described previously (Tong Nakayama Vaughan ampKanwisher 1998) Twelve 6-mm-thick near-coronal sliceswere oriented parallel to the brainstem and covered theoccipital lobe as well as the posterior portions of thetemporal and the parietal lobes One hundred sixty-eight functional images were collected for each slice ineach scan

Data Analysis

Each subjectrsquos MTMST was identified from the aver-age of the functional localizer scans as the set of allcontiguous voxels in the vicinity of the ascending limbof the inferior temporal sulcus (Tootell et al 1995bWatson et al 1993 Zeki et al 1991) that showedsignificantly stronger activation to moving comparedto static low-contrast concentric rings on a Kolmogor-ovndashSmirnov test at the level of plt0001 (uncorrected)In principle significant differences in KolmogorovndashSmirnov statistics can reflect differences in the var-iance only rather than in the means across conditions(Aguirre Zarahn amp DrsquoEsposito 1998) However thefact that the region selected by this procedure didindeed respond more strongly during the movingthan stationary conditions was confirmed by subse-quent analyses In particular t-tests across subjectsrevealed that the percent signal change in the selectedROIs was higher for moving than stationary conditions(a difference of 09 (t(7)=73 plt001) for Experi-ment 1 and 07 (t(3)=63 plt01) for Experiment 2)Moreover as shown in Figure 1 t-tests on the aver-aged group data for five subjects showed significantlystronger activation to moving compared to static rings( plt10ndash7 equivalent to plt10ndash1 after Bonferronicorrection)

For the analysis of the experimental scans the timecourse of MR signal intensity was extracted from eachsubjectrsquos MTMST by averaging the data from allvoxels within the ROI The average percent signalchange in MTMST was calculated for each subjectand stimulus type using the average signal intensityduring fixation epochs for the same subject experi-ment and task as a baseline Because the fMRIresponse typically lags four to six seconds after theneural response our data-analysis procedure treatedthe first image in each epoch as belonging to thecondition of the preceding epoch and omitted thenext two images (during the transition betweenepochs) from the analysis

An ANOVA across subjects was run on the averagepercent signal change in each of the conditions in eachexperiment Because data were analysed within inde-

Kourtzi and Kanwisher 53

pendently defined ROIs for MTMST no correction formultiple voxelwise comparisons was required

Acknowledgments

We would like to thank Ted Adelson and Maggie Shiffrar fortheir helpful comments and suggestions on this project andBruce Rosen and many people at the MGH-NMR Center fortechnical assistance and support We would also like to thankPaul Downing and Russell Epstein for their comments onprevious versions of this manuscript This research wassupported by NIMH Grant 56037 and a Human Frontiersgrant to Nancy Kanwisher Some of the results discussed in thismanuscript were first presented at the 1998 meeting of theSociety for Neuroscience in Los Angeles CA and the 1998meeting of the Psychonomic Society in Dallas TX

Reprint requests should be sent to Zoe Kourtzi Dept of Brainand Cognitive Science MIT NE20-4043 77 Massachusetts AveCambridge MA 02139-4307 or via e-mail zoepsychemitedu

REFERENCES

Aguirre G K Zarahn E amp DrsquoEsposito M (1998) A critique ofthe use of the KolmogorovndashSmirnov (KS) statistic for theanalysis of BOLD fMRI data Magnetic Resonance in Medi-cine 39 500 ndash505

Beauchamp M S Cox R W amp DeYoe E A (1997) Gradedeffects of spatial and featural attention on human-area MTand associated motion-processing areas Journal of Neuro-physiology 78 516 ndash520

Bonda E Petrides M Ostry D amp Evans A (1996) Specificinvolvement of human-parietal systems and the amygdalain the perception of biological motion The Journal ofNeuroscience 16 3737ndash 3744

Britten K H Newsome W T Shalden M N Celebrini S ampMovshon J A (1996) A relationship between behavioralchoice and the visual responses of neurons in macaque MTVisual Neuroscience 13 87ndash100

Chao L L Haxby J V Lalonde F M Ungerleider L G ampMartin A (1998) Pictures of animals and tools deferen-tially engage object form-related and motion-related brainregions Paper presented at the 28th Annual Meeting of theSociety for Neuroscience LA CA

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1990) Attentional modulation of neuralprocessing of shape color and velocity in humans Science248 1556 ndash1559

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1991) Selective and divided attention duringvisual discriminations of shape color and speed Functionalanatomy by positron emission tomography Journal ofNeuroscience 11 2383ndash2402

De Jong B M Shipp S Skidmore B Frackowiak R S Jamp Zeki S (1994) The cerebral activity related to visualperception of forward motion in depth Brain 117 1039ndash1054

Dubner R amp Zeki S M (1971) Response properties andreceptive fields of cells in an anatomically-defined region ofthe superior temporal sulcus in the monkey Brain Re-search 35 528ndash532

Dupont P Orban G A De Bruyn B Verbruggen A ampMortelmans L (1994) Many areas in the human brain re-spond to visual motion Journal of Neurophysiology 721420ndash1424

Freyd J (1983) The mental representation of movement

when static stimuli are viewed Perception and Psycho-physics 33 575ndash581

Goebel R Khorram-Sefat D Muckli L Hacker H amp SingerW (1998) The constructive nature of vision Direct evidencefrom functional magnetic resonance imaging studies of ap-parent motion and motion imagery European Journal ofNeuroscience 10 1563 ndash1573

Kaneoke Y Bundou M Koyama S Suzuki H amp Kakigi R(1997) Human cortical area responding to stimuli in ap-parent motion NeuroReport 8 677ndash 682

Martin A Haxby J V Lalonde F M Wiggs C L amp Unger-leider L G (1995) Discrete cortical regions associated withknowledge of color and knowledge of action Science 270102 ndash105

Martin A Wiggs C L Ungerleider L G amp Haxby J V(1996) Neural correlates of category-specific knowledgeNature 379 649 ndash652

Maunsell J H amp Van Essen D C (1983) The connections ofthe middle temporal-visual area (MT) and their relationshipto a cortical hierarchy in the macaque monkey Journal ofNeuroscience 3 2563 ndash2586

OrsquoCraven K M amp Kanwisher N G (1997) Visual imagery ofmoving stimuli activates area MTMST Paper presented atthe 27th Annual Meeting of the Society for NeuroscienceNew Orleans LA

OrsquoCraven K M Rosen B R Kwong K K Treisman A ampSavoy R L (1997) Voluntary attention modulates fMRI ac-tivity in human MT-MST Neuron 18 591ndash 598

Orban G A Dupont P De Bruyn B Vogels R Vanden-berghe R amp Mortelmans L (1995) A motion area in humanvisual cortex Proceedings of the National Academy ofScience 92 993 ndash997

Puce A Allison T Bentin S Gore J C amp McCarthy G(1998) Temporal-cortex activation in human subjects view-ing eye and mouth movements Journal of Neuroscience18 2188 ndash2199

Shipp S de Jong B M Zihl J Frackowiak R S J amp ZekiS (1994) The brain activity related to residual motion vi-sion in a patient with bilateral lesions of V5 Brain 1171023 ndash1038

Talairach J amp Tournoux P (1988) Co-planar stereotaxicatlas of the human brain New York Thieme Medical

Tong F Nakayama K Vaughan J T amp Kanwisher N (1998)Binocular rivalry and visual awareness in human extrastriatecortex Neuron 21 753 ndash759

Tootell R B H Mendola J D Hadjikhani N K Ledden PJ Liu A K Reppas J B Sereno M I amp Dale A M(1997) Functional analysis of V3A and related areas in hu-man visual cortex The Journal of Neuroscience 15 7060 ndash7078

Tootell R B H Reppas J B Dale A M Look R B SerenoM I Malach R Brady T J amp Rosen B R (1995a) Visualmotion after effect in human cortical area MT revealed byfunctional magnetic resonance imaging Nature 11 139ndash141

Tootell R B H Reppas J B Kwong K K Malach R BornR T Brady T J Rosen B R amp Belliveau J W (1995b)Functional analysis of human MT and related-visual corticalareas using magnetic resonance imaging The Journal ofNeuroscience 15 3215 ndash3230

Treue S amp Maunsell J H R (1996) Attentional modulationof visual motion processing in cortical areas MT and MSTNature 382 539 ndash541

Van Essen D C Maunsell J H amp Bixby J L (1981) Themiddle temporal visual area in the macaque myeloarchi-tecture connections functional properties and topographicorganization Journal of Comparative Neurology 199 293 ndash326

54 Journal of Cognitive Neuroscience Volume 12 Number 1

DISCUSSION

Our results suggest that cortical areas involved in theanalysis of physical stimulus motion can be also en-gaged automatically by static images that merely implymotion Specifically passively observed static snapshotsof objects in action activate human motion areas (MTMST) more than static images of objects withoutimplied motion These results are observed for imagesimplying animate motion such as humans or animalsin action as well as inanimate motion such as activenature scenes

It is unlikely that these results can be explained bylow-level differences among the images in the differentconditions (for example differences in the location ofthe luminance edges) The activation in MTMST wassystematically greater for implied than no implied mo-tion across eight very different stimulus categories usedin the two experiments Furthermore it is unlikely thatthe modulation of activity in MT MST is related todifferences in image flicker (each photograph was dis-played for 300 msec followed by a 500-msec blankinterval followed by the next stimulus) since this flickeroccurred in all of our stimulus conditions

These results raise numerous questions about theanalysis of object motion in the human brain That isis MTMST involved in extracting implied motion infor-mation or is it influenced by such processes occurringelsewhere in the brain It seems unlikely that theperceptual analyses involved in the inference of motionfrom still images could be computed within MTMSTNeurophysiological and imaging studies have stronglysupported the role of MTMST in the analysis of stimulusmotion but not in processes such as object recognitionInferring motion from still images depends on objectcategorization and knowledge about the repertoire ofbehavior different objects can exhibit It seems mostlikely that such high-level perceptual inferences occurelsewhere in the brain and modulate activity in MTMSTin a top-down fashion Thus the observed activationsmay reflect an expectancy of object motion that could berepresented or influence representations in areas in-volved in processing physical stimulus motion (that isMTMST)

Consistent with this hypothesis the activation forimplied vs no implied motion extended beyond MTMST to several contiguous regions as shown in Figure 1These results are consistent with recent studies suggest-ing that other areas extending posterior and superior oranterior and inferior to MTMST are also involved inmotion analysis (De Jong Shipp Skidmore Frackowiakamp Zeki 1994 Dupont et al 1994 Shipp De Jong ZihlFrackowiak amp Zeki 1994 Watson et al 1993 ) Previousresearch has shown activation anterior and medial to MTfor passive viewing of images of illusory motion (ZekiWatson amp Frackowiak 1993) tool naming (MartinWiggs Ungerleider amp Haxby 1996) and the genera-

tion of action words (Martin Haxby Lalonde Wiggs ampUngerleider 1995) Recent imaging studies have shownactivation for motion boundaries in areas V3A (Tootellet al 1997 ) and KO (Orban Dupont De Bruyn VogelsVandenberghe amp Mortelmans 1995 Van OostendeSunaert Van Hecke Marchal amp Orban 1997 ) extendingposterior and medial to MT along the occipital surfaceThe activations observed in our subjects in the vicinity ofthe superior temporal sulcus are also consistent withprevious studies showing activation in the superiortemporal sulcus for motion imagery (Goebel et al1998) and viewing of biological motion stimuli (BondaPetrides Ostry amp Evans 1996 Puce Allison BentinGore amp McCarthy 1998)

Finally several prior findings support the hypothesisthat the current results reflect top-down influences ofhigh-level perceptual inferences on MTMST Both sin-gle unit (Treue amp Maunsell 1996 ) and fMRI studies(Beauchamp Cox amp DeYoe 1997 Corbetta MiezinDobmeyer Shulman amp Petersen 1990 1991 OrsquoCravenRosen Kwong Treisman amp Savoy 1997) have demon-strated that the response of MTMST to moving stimulican be strongly modulated by visual attention Alsoactivity in MTMST has been demonstrated even whensubjects close their eyes and merely imagine movingcompared to stationary arrays (Goebel et al 1998OrsquoCraven amp Kanwisher 1997 )

While the present work is consistent with these pre-vious studies suggesting that activation in MTMST canbe modulated in a top-down fashion we show here forthe first time that such top-down effects can occurautomatically That is dynamic information implicit inthe image was extracted and influenced activity in MTMST even though subjects were not asked or requiredto perceive attend to or imagine motion

One possible interpretation of our findings is thatinferring motion may involve or result in motion ima-gery Another interpretation is that the processing of aparticular object category (for example animals) maylead to activation of regions involved in processingproperties highly associated with that object category(for example motion) (Chao Haxby Lalonde Ungerlei-der amp Martin 1998 Martin et al 1996) Consistent withthe second hypothesis the current findings show thatactivation in MTMST is significantly higher for images ofpeople even people at rest than for images of houses

More broadly the current results support an emer-ging view of extrastriate cortex as playing a crucial rolenot only in visual perception but also in visual cogni-tion

METHODS

Subjects

Ten right-handed MIT students participated in Experi-ment 1 four in the passive viewing condition and six inthe 1-back matching condition Two subjects tested on

52 Journal of Cognitive Neuroscience Volume 12 Number 1

the 1-back matching condition were excluded from theanalysis due to excessive head motion Another six right-handed MIT students participated in Experiment 2 Twosubjects were excluded from the analysis in this condi-tion due to excessive head motion

Materials and Design

The stimuli used for functionally localizing MT were lowcontrast moving vs stationary concentric rings as de-scribed in Tootell et al (1995a) For the experimentalconditions stimuli were 300pound300 pixel digitized grays-cale photographs Experiment 1 involved a mixed de-sign with Stimulus Type a within-subject variable (withfour levels photographs of athletes with implied mo-tion athletes without implied motion people at restand houses) and Task a between-subjects factor (withtwo levels passive viewing vs 1-back repetition detec-tion) Experiment 2 involved two orthogonal factorscrossed within subjects Stimulus Type (animals vsscenes) and Condition (implied motion vs no impliedmotion)

Procedure

Each subject was run on two or more functional MTlocalizer scans with low contrast moving vs stationaryconcentric rings (as described in Tootell et al 1995a)Then each subject was run on four scans of the experi-mental test materials For the passive viewing condi-tions the subjects were asked to observe the imagescarefully while fixating a dot in the center of the image(Monitoring of eye movements outside the scanner forthree subjects that participated in Experiment 1 andthree subjects that participated in Experiment 2 showedthat the number of eye movements was very small in allconditions and did not differ significantly across condi-tions) For the 1-back matching condition subjects wereinstructed to press a button whenever they saw twoidentical pictures in a row Two or more repetitionsoccurred in each epoch

Each scan lasted 5 min and 36 sec and consisted ofsixteen 16-sec epochs with fixation periods interleavedas shown in Figures 2 and 3 Twenty different photo-graphs of the same type were presented in each epochEach photograph was presented for 300 msec with ablank interval of 500 msec between photographs Eachof the four stimulus types in each experiment werepresented in four different epochs within each scan ina design that balanced for the order of conditions asshown in Figures 2 and 3

MRI Acquisition

Scanning was done on the 3 T scanner (modified byANMR for Echo Planar Imaging) at the MGH-NMRCenter in Charlestown MA A custom bilateral surface

coil (built by J Thomas Vaughan) provided a high signal-to-noise ratio in posterior brain regions A bite-bar wasused to minimize head motion Standard imaging pro-cedures (Gradient Echo pulse sequence TR 2 sec TE30 msec flip angle 908 1808 offset 25 msec) were usedas described previously (Tong Nakayama Vaughan ampKanwisher 1998) Twelve 6-mm-thick near-coronal sliceswere oriented parallel to the brainstem and covered theoccipital lobe as well as the posterior portions of thetemporal and the parietal lobes One hundred sixty-eight functional images were collected for each slice ineach scan

Data Analysis

Each subjectrsquos MTMST was identified from the aver-age of the functional localizer scans as the set of allcontiguous voxels in the vicinity of the ascending limbof the inferior temporal sulcus (Tootell et al 1995bWatson et al 1993 Zeki et al 1991) that showedsignificantly stronger activation to moving comparedto static low-contrast concentric rings on a Kolmogor-ovndashSmirnov test at the level of plt0001 (uncorrected)In principle significant differences in KolmogorovndashSmirnov statistics can reflect differences in the var-iance only rather than in the means across conditions(Aguirre Zarahn amp DrsquoEsposito 1998) However thefact that the region selected by this procedure didindeed respond more strongly during the movingthan stationary conditions was confirmed by subse-quent analyses In particular t-tests across subjectsrevealed that the percent signal change in the selectedROIs was higher for moving than stationary conditions(a difference of 09 (t(7)=73 plt001) for Experi-ment 1 and 07 (t(3)=63 plt01) for Experiment 2)Moreover as shown in Figure 1 t-tests on the aver-aged group data for five subjects showed significantlystronger activation to moving compared to static rings( plt10ndash7 equivalent to plt10ndash1 after Bonferronicorrection)

For the analysis of the experimental scans the timecourse of MR signal intensity was extracted from eachsubjectrsquos MTMST by averaging the data from allvoxels within the ROI The average percent signalchange in MTMST was calculated for each subjectand stimulus type using the average signal intensityduring fixation epochs for the same subject experi-ment and task as a baseline Because the fMRIresponse typically lags four to six seconds after theneural response our data-analysis procedure treatedthe first image in each epoch as belonging to thecondition of the preceding epoch and omitted thenext two images (during the transition betweenepochs) from the analysis

An ANOVA across subjects was run on the averagepercent signal change in each of the conditions in eachexperiment Because data were analysed within inde-

Kourtzi and Kanwisher 53

pendently defined ROIs for MTMST no correction formultiple voxelwise comparisons was required

Acknowledgments

We would like to thank Ted Adelson and Maggie Shiffrar fortheir helpful comments and suggestions on this project andBruce Rosen and many people at the MGH-NMR Center fortechnical assistance and support We would also like to thankPaul Downing and Russell Epstein for their comments onprevious versions of this manuscript This research wassupported by NIMH Grant 56037 and a Human Frontiersgrant to Nancy Kanwisher Some of the results discussed in thismanuscript were first presented at the 1998 meeting of theSociety for Neuroscience in Los Angeles CA and the 1998meeting of the Psychonomic Society in Dallas TX

Reprint requests should be sent to Zoe Kourtzi Dept of Brainand Cognitive Science MIT NE20-4043 77 Massachusetts AveCambridge MA 02139-4307 or via e-mail zoepsychemitedu

REFERENCES

Aguirre G K Zarahn E amp DrsquoEsposito M (1998) A critique ofthe use of the KolmogorovndashSmirnov (KS) statistic for theanalysis of BOLD fMRI data Magnetic Resonance in Medi-cine 39 500 ndash505

Beauchamp M S Cox R W amp DeYoe E A (1997) Gradedeffects of spatial and featural attention on human-area MTand associated motion-processing areas Journal of Neuro-physiology 78 516 ndash520

Bonda E Petrides M Ostry D amp Evans A (1996) Specificinvolvement of human-parietal systems and the amygdalain the perception of biological motion The Journal ofNeuroscience 16 3737ndash 3744

Britten K H Newsome W T Shalden M N Celebrini S ampMovshon J A (1996) A relationship between behavioralchoice and the visual responses of neurons in macaque MTVisual Neuroscience 13 87ndash100

Chao L L Haxby J V Lalonde F M Ungerleider L G ampMartin A (1998) Pictures of animals and tools deferen-tially engage object form-related and motion-related brainregions Paper presented at the 28th Annual Meeting of theSociety for Neuroscience LA CA

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1990) Attentional modulation of neuralprocessing of shape color and velocity in humans Science248 1556 ndash1559

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1991) Selective and divided attention duringvisual discriminations of shape color and speed Functionalanatomy by positron emission tomography Journal ofNeuroscience 11 2383ndash2402

De Jong B M Shipp S Skidmore B Frackowiak R S Jamp Zeki S (1994) The cerebral activity related to visualperception of forward motion in depth Brain 117 1039ndash1054

Dubner R amp Zeki S M (1971) Response properties andreceptive fields of cells in an anatomically-defined region ofthe superior temporal sulcus in the monkey Brain Re-search 35 528ndash532

Dupont P Orban G A De Bruyn B Verbruggen A ampMortelmans L (1994) Many areas in the human brain re-spond to visual motion Journal of Neurophysiology 721420ndash1424

Freyd J (1983) The mental representation of movement

when static stimuli are viewed Perception and Psycho-physics 33 575ndash581

Goebel R Khorram-Sefat D Muckli L Hacker H amp SingerW (1998) The constructive nature of vision Direct evidencefrom functional magnetic resonance imaging studies of ap-parent motion and motion imagery European Journal ofNeuroscience 10 1563 ndash1573

Kaneoke Y Bundou M Koyama S Suzuki H amp Kakigi R(1997) Human cortical area responding to stimuli in ap-parent motion NeuroReport 8 677ndash 682

Martin A Haxby J V Lalonde F M Wiggs C L amp Unger-leider L G (1995) Discrete cortical regions associated withknowledge of color and knowledge of action Science 270102 ndash105

Martin A Wiggs C L Ungerleider L G amp Haxby J V(1996) Neural correlates of category-specific knowledgeNature 379 649 ndash652

Maunsell J H amp Van Essen D C (1983) The connections ofthe middle temporal-visual area (MT) and their relationshipto a cortical hierarchy in the macaque monkey Journal ofNeuroscience 3 2563 ndash2586

OrsquoCraven K M amp Kanwisher N G (1997) Visual imagery ofmoving stimuli activates area MTMST Paper presented atthe 27th Annual Meeting of the Society for NeuroscienceNew Orleans LA

OrsquoCraven K M Rosen B R Kwong K K Treisman A ampSavoy R L (1997) Voluntary attention modulates fMRI ac-tivity in human MT-MST Neuron 18 591ndash 598

Orban G A Dupont P De Bruyn B Vogels R Vanden-berghe R amp Mortelmans L (1995) A motion area in humanvisual cortex Proceedings of the National Academy ofScience 92 993 ndash997

Puce A Allison T Bentin S Gore J C amp McCarthy G(1998) Temporal-cortex activation in human subjects view-ing eye and mouth movements Journal of Neuroscience18 2188 ndash2199

Shipp S de Jong B M Zihl J Frackowiak R S J amp ZekiS (1994) The brain activity related to residual motion vi-sion in a patient with bilateral lesions of V5 Brain 1171023 ndash1038

Talairach J amp Tournoux P (1988) Co-planar stereotaxicatlas of the human brain New York Thieme Medical

Tong F Nakayama K Vaughan J T amp Kanwisher N (1998)Binocular rivalry and visual awareness in human extrastriatecortex Neuron 21 753 ndash759

Tootell R B H Mendola J D Hadjikhani N K Ledden PJ Liu A K Reppas J B Sereno M I amp Dale A M(1997) Functional analysis of V3A and related areas in hu-man visual cortex The Journal of Neuroscience 15 7060 ndash7078

Tootell R B H Reppas J B Dale A M Look R B SerenoM I Malach R Brady T J amp Rosen B R (1995a) Visualmotion after effect in human cortical area MT revealed byfunctional magnetic resonance imaging Nature 11 139ndash141

Tootell R B H Reppas J B Kwong K K Malach R BornR T Brady T J Rosen B R amp Belliveau J W (1995b)Functional analysis of human MT and related-visual corticalareas using magnetic resonance imaging The Journal ofNeuroscience 15 3215 ndash3230

Treue S amp Maunsell J H R (1996) Attentional modulationof visual motion processing in cortical areas MT and MSTNature 382 539 ndash541

Van Essen D C Maunsell J H amp Bixby J L (1981) Themiddle temporal visual area in the macaque myeloarchi-tecture connections functional properties and topographicorganization Journal of Comparative Neurology 199 293 ndash326

54 Journal of Cognitive Neuroscience Volume 12 Number 1

the 1-back matching condition were excluded from theanalysis due to excessive head motion Another six right-handed MIT students participated in Experiment 2 Twosubjects were excluded from the analysis in this condi-tion due to excessive head motion

Materials and Design

The stimuli used for functionally localizing MT were lowcontrast moving vs stationary concentric rings as de-scribed in Tootell et al (1995a) For the experimentalconditions stimuli were 300pound300 pixel digitized grays-cale photographs Experiment 1 involved a mixed de-sign with Stimulus Type a within-subject variable (withfour levels photographs of athletes with implied mo-tion athletes without implied motion people at restand houses) and Task a between-subjects factor (withtwo levels passive viewing vs 1-back repetition detec-tion) Experiment 2 involved two orthogonal factorscrossed within subjects Stimulus Type (animals vsscenes) and Condition (implied motion vs no impliedmotion)

Procedure

Each subject was run on two or more functional MTlocalizer scans with low contrast moving vs stationaryconcentric rings (as described in Tootell et al 1995a)Then each subject was run on four scans of the experi-mental test materials For the passive viewing condi-tions the subjects were asked to observe the imagescarefully while fixating a dot in the center of the image(Monitoring of eye movements outside the scanner forthree subjects that participated in Experiment 1 andthree subjects that participated in Experiment 2 showedthat the number of eye movements was very small in allconditions and did not differ significantly across condi-tions) For the 1-back matching condition subjects wereinstructed to press a button whenever they saw twoidentical pictures in a row Two or more repetitionsoccurred in each epoch

Each scan lasted 5 min and 36 sec and consisted ofsixteen 16-sec epochs with fixation periods interleavedas shown in Figures 2 and 3 Twenty different photo-graphs of the same type were presented in each epochEach photograph was presented for 300 msec with ablank interval of 500 msec between photographs Eachof the four stimulus types in each experiment werepresented in four different epochs within each scan ina design that balanced for the order of conditions asshown in Figures 2 and 3

MRI Acquisition

Scanning was done on the 3 T scanner (modified byANMR for Echo Planar Imaging) at the MGH-NMRCenter in Charlestown MA A custom bilateral surface

coil (built by J Thomas Vaughan) provided a high signal-to-noise ratio in posterior brain regions A bite-bar wasused to minimize head motion Standard imaging pro-cedures (Gradient Echo pulse sequence TR 2 sec TE30 msec flip angle 908 1808 offset 25 msec) were usedas described previously (Tong Nakayama Vaughan ampKanwisher 1998) Twelve 6-mm-thick near-coronal sliceswere oriented parallel to the brainstem and covered theoccipital lobe as well as the posterior portions of thetemporal and the parietal lobes One hundred sixty-eight functional images were collected for each slice ineach scan

Data Analysis

Each subjectrsquos MTMST was identified from the aver-age of the functional localizer scans as the set of allcontiguous voxels in the vicinity of the ascending limbof the inferior temporal sulcus (Tootell et al 1995bWatson et al 1993 Zeki et al 1991) that showedsignificantly stronger activation to moving comparedto static low-contrast concentric rings on a Kolmogor-ovndashSmirnov test at the level of plt0001 (uncorrected)In principle significant differences in KolmogorovndashSmirnov statistics can reflect differences in the var-iance only rather than in the means across conditions(Aguirre Zarahn amp DrsquoEsposito 1998) However thefact that the region selected by this procedure didindeed respond more strongly during the movingthan stationary conditions was confirmed by subse-quent analyses In particular t-tests across subjectsrevealed that the percent signal change in the selectedROIs was higher for moving than stationary conditions(a difference of 09 (t(7)=73 plt001) for Experi-ment 1 and 07 (t(3)=63 plt01) for Experiment 2)Moreover as shown in Figure 1 t-tests on the aver-aged group data for five subjects showed significantlystronger activation to moving compared to static rings( plt10ndash7 equivalent to plt10ndash1 after Bonferronicorrection)

For the analysis of the experimental scans the timecourse of MR signal intensity was extracted from eachsubjectrsquos MTMST by averaging the data from allvoxels within the ROI The average percent signalchange in MTMST was calculated for each subjectand stimulus type using the average signal intensityduring fixation epochs for the same subject experi-ment and task as a baseline Because the fMRIresponse typically lags four to six seconds after theneural response our data-analysis procedure treatedthe first image in each epoch as belonging to thecondition of the preceding epoch and omitted thenext two images (during the transition betweenepochs) from the analysis

An ANOVA across subjects was run on the averagepercent signal change in each of the conditions in eachexperiment Because data were analysed within inde-

Kourtzi and Kanwisher 53

pendently defined ROIs for MTMST no correction formultiple voxelwise comparisons was required

Acknowledgments

We would like to thank Ted Adelson and Maggie Shiffrar fortheir helpful comments and suggestions on this project andBruce Rosen and many people at the MGH-NMR Center fortechnical assistance and support We would also like to thankPaul Downing and Russell Epstein for their comments onprevious versions of this manuscript This research wassupported by NIMH Grant 56037 and a Human Frontiersgrant to Nancy Kanwisher Some of the results discussed in thismanuscript were first presented at the 1998 meeting of theSociety for Neuroscience in Los Angeles CA and the 1998meeting of the Psychonomic Society in Dallas TX

Reprint requests should be sent to Zoe Kourtzi Dept of Brainand Cognitive Science MIT NE20-4043 77 Massachusetts AveCambridge MA 02139-4307 or via e-mail zoepsychemitedu

REFERENCES

Aguirre G K Zarahn E amp DrsquoEsposito M (1998) A critique ofthe use of the KolmogorovndashSmirnov (KS) statistic for theanalysis of BOLD fMRI data Magnetic Resonance in Medi-cine 39 500 ndash505

Beauchamp M S Cox R W amp DeYoe E A (1997) Gradedeffects of spatial and featural attention on human-area MTand associated motion-processing areas Journal of Neuro-physiology 78 516 ndash520

Bonda E Petrides M Ostry D amp Evans A (1996) Specificinvolvement of human-parietal systems and the amygdalain the perception of biological motion The Journal ofNeuroscience 16 3737ndash 3744

Britten K H Newsome W T Shalden M N Celebrini S ampMovshon J A (1996) A relationship between behavioralchoice and the visual responses of neurons in macaque MTVisual Neuroscience 13 87ndash100

Chao L L Haxby J V Lalonde F M Ungerleider L G ampMartin A (1998) Pictures of animals and tools deferen-tially engage object form-related and motion-related brainregions Paper presented at the 28th Annual Meeting of theSociety for Neuroscience LA CA

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1990) Attentional modulation of neuralprocessing of shape color and velocity in humans Science248 1556 ndash1559

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1991) Selective and divided attention duringvisual discriminations of shape color and speed Functionalanatomy by positron emission tomography Journal ofNeuroscience 11 2383ndash2402

De Jong B M Shipp S Skidmore B Frackowiak R S Jamp Zeki S (1994) The cerebral activity related to visualperception of forward motion in depth Brain 117 1039ndash1054

Dubner R amp Zeki S M (1971) Response properties andreceptive fields of cells in an anatomically-defined region ofthe superior temporal sulcus in the monkey Brain Re-search 35 528ndash532

Dupont P Orban G A De Bruyn B Verbruggen A ampMortelmans L (1994) Many areas in the human brain re-spond to visual motion Journal of Neurophysiology 721420ndash1424

Freyd J (1983) The mental representation of movement

when static stimuli are viewed Perception and Psycho-physics 33 575ndash581

Goebel R Khorram-Sefat D Muckli L Hacker H amp SingerW (1998) The constructive nature of vision Direct evidencefrom functional magnetic resonance imaging studies of ap-parent motion and motion imagery European Journal ofNeuroscience 10 1563 ndash1573

Kaneoke Y Bundou M Koyama S Suzuki H amp Kakigi R(1997) Human cortical area responding to stimuli in ap-parent motion NeuroReport 8 677ndash 682

Martin A Haxby J V Lalonde F M Wiggs C L amp Unger-leider L G (1995) Discrete cortical regions associated withknowledge of color and knowledge of action Science 270102 ndash105

Martin A Wiggs C L Ungerleider L G amp Haxby J V(1996) Neural correlates of category-specific knowledgeNature 379 649 ndash652

Maunsell J H amp Van Essen D C (1983) The connections ofthe middle temporal-visual area (MT) and their relationshipto a cortical hierarchy in the macaque monkey Journal ofNeuroscience 3 2563 ndash2586

OrsquoCraven K M amp Kanwisher N G (1997) Visual imagery ofmoving stimuli activates area MTMST Paper presented atthe 27th Annual Meeting of the Society for NeuroscienceNew Orleans LA

OrsquoCraven K M Rosen B R Kwong K K Treisman A ampSavoy R L (1997) Voluntary attention modulates fMRI ac-tivity in human MT-MST Neuron 18 591ndash 598

Orban G A Dupont P De Bruyn B Vogels R Vanden-berghe R amp Mortelmans L (1995) A motion area in humanvisual cortex Proceedings of the National Academy ofScience 92 993 ndash997

Puce A Allison T Bentin S Gore J C amp McCarthy G(1998) Temporal-cortex activation in human subjects view-ing eye and mouth movements Journal of Neuroscience18 2188 ndash2199

Shipp S de Jong B M Zihl J Frackowiak R S J amp ZekiS (1994) The brain activity related to residual motion vi-sion in a patient with bilateral lesions of V5 Brain 1171023 ndash1038

Talairach J amp Tournoux P (1988) Co-planar stereotaxicatlas of the human brain New York Thieme Medical

Tong F Nakayama K Vaughan J T amp Kanwisher N (1998)Binocular rivalry and visual awareness in human extrastriatecortex Neuron 21 753 ndash759

Tootell R B H Mendola J D Hadjikhani N K Ledden PJ Liu A K Reppas J B Sereno M I amp Dale A M(1997) Functional analysis of V3A and related areas in hu-man visual cortex The Journal of Neuroscience 15 7060 ndash7078

Tootell R B H Reppas J B Dale A M Look R B SerenoM I Malach R Brady T J amp Rosen B R (1995a) Visualmotion after effect in human cortical area MT revealed byfunctional magnetic resonance imaging Nature 11 139ndash141

Tootell R B H Reppas J B Kwong K K Malach R BornR T Brady T J Rosen B R amp Belliveau J W (1995b)Functional analysis of human MT and related-visual corticalareas using magnetic resonance imaging The Journal ofNeuroscience 15 3215 ndash3230

Treue S amp Maunsell J H R (1996) Attentional modulationof visual motion processing in cortical areas MT and MSTNature 382 539 ndash541

Van Essen D C Maunsell J H amp Bixby J L (1981) Themiddle temporal visual area in the macaque myeloarchi-tecture connections functional properties and topographicorganization Journal of Comparative Neurology 199 293 ndash326

54 Journal of Cognitive Neuroscience Volume 12 Number 1

pendently defined ROIs for MTMST no correction formultiple voxelwise comparisons was required

Acknowledgments

We would like to thank Ted Adelson and Maggie Shiffrar fortheir helpful comments and suggestions on this project andBruce Rosen and many people at the MGH-NMR Center fortechnical assistance and support We would also like to thankPaul Downing and Russell Epstein for their comments onprevious versions of this manuscript This research wassupported by NIMH Grant 56037 and a Human Frontiersgrant to Nancy Kanwisher Some of the results discussed in thismanuscript were first presented at the 1998 meeting of theSociety for Neuroscience in Los Angeles CA and the 1998meeting of the Psychonomic Society in Dallas TX

Reprint requests should be sent to Zoe Kourtzi Dept of Brainand Cognitive Science MIT NE20-4043 77 Massachusetts AveCambridge MA 02139-4307 or via e-mail zoepsychemitedu

REFERENCES

Aguirre G K Zarahn E amp DrsquoEsposito M (1998) A critique ofthe use of the KolmogorovndashSmirnov (KS) statistic for theanalysis of BOLD fMRI data Magnetic Resonance in Medi-cine 39 500 ndash505

Beauchamp M S Cox R W amp DeYoe E A (1997) Gradedeffects of spatial and featural attention on human-area MTand associated motion-processing areas Journal of Neuro-physiology 78 516 ndash520

Bonda E Petrides M Ostry D amp Evans A (1996) Specificinvolvement of human-parietal systems and the amygdalain the perception of biological motion The Journal ofNeuroscience 16 3737ndash 3744

Britten K H Newsome W T Shalden M N Celebrini S ampMovshon J A (1996) A relationship between behavioralchoice and the visual responses of neurons in macaque MTVisual Neuroscience 13 87ndash100

Chao L L Haxby J V Lalonde F M Ungerleider L G ampMartin A (1998) Pictures of animals and tools deferen-tially engage object form-related and motion-related brainregions Paper presented at the 28th Annual Meeting of theSociety for Neuroscience LA CA

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1990) Attentional modulation of neuralprocessing of shape color and velocity in humans Science248 1556 ndash1559

Corbetta M Miezin F M Dobmeyer S Shulman G L ampPetersen S E (1991) Selective and divided attention duringvisual discriminations of shape color and speed Functionalanatomy by positron emission tomography Journal ofNeuroscience 11 2383ndash2402

De Jong B M Shipp S Skidmore B Frackowiak R S Jamp Zeki S (1994) The cerebral activity related to visualperception of forward motion in depth Brain 117 1039ndash1054

Dubner R amp Zeki S M (1971) Response properties andreceptive fields of cells in an anatomically-defined region ofthe superior temporal sulcus in the monkey Brain Re-search 35 528ndash532

Dupont P Orban G A De Bruyn B Verbruggen A ampMortelmans L (1994) Many areas in the human brain re-spond to visual motion Journal of Neurophysiology 721420ndash1424

Freyd J (1983) The mental representation of movement

when static stimuli are viewed Perception and Psycho-physics 33 575ndash581

Goebel R Khorram-Sefat D Muckli L Hacker H amp SingerW (1998) The constructive nature of vision Direct evidencefrom functional magnetic resonance imaging studies of ap-parent motion and motion imagery European Journal ofNeuroscience 10 1563 ndash1573

Kaneoke Y Bundou M Koyama S Suzuki H amp Kakigi R(1997) Human cortical area responding to stimuli in ap-parent motion NeuroReport 8 677ndash 682

Martin A Haxby J V Lalonde F M Wiggs C L amp Unger-leider L G (1995) Discrete cortical regions associated withknowledge of color and knowledge of action Science 270102 ndash105

Martin A Wiggs C L Ungerleider L G amp Haxby J V(1996) Neural correlates of category-specific knowledgeNature 379 649 ndash652

Maunsell J H amp Van Essen D C (1983) The connections ofthe middle temporal-visual area (MT) and their relationshipto a cortical hierarchy in the macaque monkey Journal ofNeuroscience 3 2563 ndash2586

OrsquoCraven K M amp Kanwisher N G (1997) Visual imagery ofmoving stimuli activates area MTMST Paper presented atthe 27th Annual Meeting of the Society for NeuroscienceNew Orleans LA

OrsquoCraven K M Rosen B R Kwong K K Treisman A ampSavoy R L (1997) Voluntary attention modulates fMRI ac-tivity in human MT-MST Neuron 18 591ndash 598

Orban G A Dupont P De Bruyn B Vogels R Vanden-berghe R amp Mortelmans L (1995) A motion area in humanvisual cortex Proceedings of the National Academy ofScience 92 993 ndash997

Puce A Allison T Bentin S Gore J C amp McCarthy G(1998) Temporal-cortex activation in human subjects view-ing eye and mouth movements Journal of Neuroscience18 2188 ndash2199

Shipp S de Jong B M Zihl J Frackowiak R S J amp ZekiS (1994) The brain activity related to residual motion vi-sion in a patient with bilateral lesions of V5 Brain 1171023 ndash1038

Talairach J amp Tournoux P (1988) Co-planar stereotaxicatlas of the human brain New York Thieme Medical

Tong F Nakayama K Vaughan J T amp Kanwisher N (1998)Binocular rivalry and visual awareness in human extrastriatecortex Neuron 21 753 ndash759

Tootell R B H Mendola J D Hadjikhani N K Ledden PJ Liu A K Reppas J B Sereno M I amp Dale A M(1997) Functional analysis of V3A and related areas in hu-man visual cortex The Journal of Neuroscience 15 7060 ndash7078

Tootell R B H Reppas J B Dale A M Look R B SerenoM I Malach R Brady T J amp Rosen B R (1995a) Visualmotion after effect in human cortical area MT revealed byfunctional magnetic resonance imaging Nature 11 139ndash141

Tootell R B H Reppas J B Kwong K K Malach R BornR T Brady T J Rosen B R amp Belliveau J W (1995b)Functional analysis of human MT and related-visual corticalareas using magnetic resonance imaging The Journal ofNeuroscience 15 3215 ndash3230

Treue S amp Maunsell J H R (1996) Attentional modulationof visual motion processing in cortical areas MT and MSTNature 382 539 ndash541

Van Essen D C Maunsell J H amp Bixby J L (1981) Themiddle temporal visual area in the macaque myeloarchi-tecture connections functional properties and topographicorganization Journal of Comparative Neurology 199 293 ndash326

54 Journal of Cognitive Neuroscience Volume 12 Number 1

Van Oostende S Sunaert S Van Hecke P Marchal G ampOrban G A (1997) The Kinetic Occipital (KO) region inman An fMRI study Cerebral Cortex 7 690ndash701

Watson J D G Myers R Frackowiak R S J Hajnal J VWoods R P Mazziotta J C Shipp S amp Zeki S (1993)Area V5 of the human brain Evidence from a combinedstudy using positron emission tomography and magneticresonance imaging Cerebral Cortex 3 79ndash 94

Zeki S Watson J D G Lueck C J Friston K J KennardC amp Frackowiak R S J (1991) A direct discrimination offunctional specialization in human visual cortex Journal ofNeuroscience 11 641ndash 649

Zeki S Watson J D G amp Frackowiak R S J (1993) Goingbeyond the information given The relation of illusory visualmotion to brain activity Proceedings of the Royal Society ofLondon B 252 215 ndash222

Kourtzi and Kanwisher 55