MEMORY TRACES FOR WORDS AS REVEALED€¦  · Web viewMultimodal Processing Of Color And Form...

35
MirrorBot IST-2001-35282 Biomimetic multimodal learning in a mirror neuron-based robot Multimodal Processing Of Color And Form Information In Left Fronto-Temporal Cortex Friedemann Pulvermüller and Olaf Hauk MirrorBot Report 19b Report Version: 0 Report Preparation Date: 7 March 2022 Classification: Draft Contract Start Date: 1 st June 2002 Duration: Three Years Project Co-ordinator: Professor Stefan Wermter Partners: University of Sunderland, Institut National de Recherche en Informatique et en Automatique, Universität Ulm, Medical Research Council, Universita degli Studi di Parma

Transcript of MEMORY TRACES FOR WORDS AS REVEALED€¦  · Web viewMultimodal Processing Of Color And Form...

MirrorBot

IST-2001-35282

Biomimetic multimodal learning in a mirror neuron-based robot

Multimodal Processing Of Color And Form Information In Left Fronto-Temporal Cortex

Friedemann Pulvermüller and Olaf Hauk

MirrorBot Report 19b

Report Version: 0

Report Preparation Date: 9 May 2023

Classification: Draft

Contract Start Date: 1st June 2002 Duration: Three Years

Project Co-ordinator: Professor Stefan Wermter

Partners: University of Sunderland, Institut National de Recherche en Informatique et en Automatique, Universität Ulm, Medical Research Council, Universita degli Studi di Parma

Project funded by the European Community under the “Information Society Technologies Programme”

0 Table of Content

0 TABLE OF CONTENT.........................................................................................................................................21. ABSTRACT..........................................................................................................................................................32. INTRODUCTION.................................................................................................................................................43. EXPERIMENTAL PROCEDURES.....................................................................................................................7

3.1 IMAGING METHODS...................................................................................................................................73.2 STIMULI AND EXPERIMENTAL DESIGN................................................................................................8

4. RESULTS..............................................................................................................................................................95. DISCUSSION.....................................................................................................................................................14

5.1 GENERAL WORD RELATED ACTIVATION..........................................................................................145.2 CATEGORY SPECIFIC SEMANTIC ACTIVATION................................................................................155.3 SPECIFIC ACTIVATION IN TEMPORAL LOBE.....................................................................................155.4 SPECIFIC ACTIVATION IN FRONTAL LOBE........................................................................................16

6. CONCLUSION...................................................................................................................................................187. REFERENCES....................................................................................................................................................19

2

1. Abstract

To investigate the cortical basis of color and form concepts, we examined event-related fMRI responses to matched words related to abstract color and form information. Silent word reading elicited activity in left temporal and frontal cortex, where also category-specific activity differences were observed. Whereas color words preferentially activated anterior parahippocampal gyrus, form words evoked category-specific activity in fusiform and middle temporal gyrus as well as premotor and dorsolateral prefrontal areas in inferior and middle frontal gyri. These results demonstrate that word meanings and concepts are not processed by a unique cortical area, but by different sets of areas each of which may contribute differentially to conceptual semantic processing. We hypothesize that the anterior parahippocampal activation to color words indexes computation of visual feature conjunctions and disjunctions necessary for classifying visual stimuli under a color concept. The predominant premotor and prefrontal activation to form words suggests action-related information processing and may reflect the involvement of neuronal elements responding in an either-or fashion to mirror neurons related to adumbrating shapes.

3

2. Introduction

A great challenge in cognitive neuroscience is to define the cortical areas engaged in processing semantic meaning of concepts and words. Many researchers agree that temporal cortex provides a unique substrate for semantic binding. Definitions of relevant areas within the temporal lobe range from temporal pole (Hodges, 2001; Rogers et al., 2004) to anterior superior temporal gyrus (Scott and Johnsrude, 2003), posterior inferior temporal areas (Price et al., 2001) and posterior superior temporal sulcus (Hickok and Poeppel, 2000). Activation in temporal cortex, including posterior fusiform and parahippocampal gyri, was found to be specific to categories of knowledge (Chao et al., 1999; O'Craven and Kanwisher, 2000), suggesting that areas of temporal cortex may contribute differentially to semantic category processing.

A different line of research emphasizes the relevance of frontal cortex, especially premotor and inferior frontal cortex, for meaning processing (Posner and Pavese, 1998; Rizzolatti et al., 2001). Mirror neurons that bind information about actions and their corresponding perceptual patterns have been discovered in inferior premotor cortex of monkeys and apparently contribute to action-related cognitive processes, including observation or imagination of actions (Buccino et al., 2001; Jeannerod and Frak, 1999). Processing action verbs and tool words with semantic relationship to actions was found to elicit stronger fronto-central cortical activation than object words (Martin et al., 1996; Preissl et al., 1995; Pulvermüller et al., 1999). Among the action words, those related to movements of the face, arm or leg activated fronto-central cortex in a somatotopic fashion (Hauk et al., 2004; Shtyrov et al., 2004) consistent with the claim that sensorimotor cortex processes action-related aspects of word meaning (Pulvermüller, 2001).

These results can be explained by the so-called sensori-motor theory of conceptual and semantic processing: Sensory and action-related semantic features are attached to a symbol by correlation, typically when a child perceives words in the context of specific objects and actions (Freud, 1891; Humphreys and Forde, 2001; Kiefer and Spitzer, 2001; Shallice, 1988; Warrington and Shallice, 1984; Wernicke, 1874). Because perceptions through different senses and actions are processed by different areas of cortex, this explains a dissociation into categories of knowledge, for example visually-related and motor categories (Warrington and Shallice, 1984), and even fine-grained semantic categories, such as face-, arm- and leg-action words (Pulvermüller, 2001). Although the sensory-motor theory explains a range of neuropsychological dissociations in patients with disease of the brain (Humphreys and Forde, 2001) and a multitude of facts revealed by neuroimaging experiments (Martin and Chao, 2001; Pulvermüller, 1999) it has systematic difficulty explaining abstract knowledge (Grossman et al., 2002; Jackendoff, 2002). Words such as “rhomb”, “hope”, and “nevertheless” do not refer to objects in the world but have abstract meaning and the brain areas where such abstract knowledge is stored are still largely unknown.

Neuroimaging work comparing symbols with abstract and concrete meaning has shed light on the issue, but could not determine an area, or set of areas, consistently activated by abstract meaning processing. Some results suggested differential laterality: Highly abstract grammatical function words such as “it” and “the” led to pronounced left-lateralized

4

neurophysiological activity, which contrasted with the more bilateral responses to concrete nouns and verbs (Neville et al., 1992; Pulvermüller et al., 1995). Also, more strongly left-lateralized brain responses to abstract nouns compared with concrete nouns were revealed by ERP experiments (Kounios and Holcomb, 1994). However, a general difference in laterality related to abstractness could not be confirmed (Friederici et al., 2000). Recent imaging work using fMRI indicated left-hemispheric correlates of abstract semantics in Broca’s area (Fiebach and Friederici, 2004), angular gyrus (Skosnik et al., 2002), superior (Wise et al., 2000) and posterior temporal cortex (Grossman et al., 2002), or a widespread set of areas including temporal pole along with inferior temporal and inferior frontal cortex (Noppeney and Price, 2004). In contrast with previous findings of a predominantly left-hemispheric locus of abstract semantics, a few studies also reported stronger activation for abstract words relative to concrete words in different areas of the right non-dominant hemisphere (Grossman et al., 2002; Kiehl et al., 1999; Perani et al., 1999). However, consistent with the differential laterality hypothesis, a recent study by Binder and colleagues reported that concrete word identification led to activation of distributed bilateral sources whereas abstract words activated a left inferior frontal region including areas in the inferior and middle frontal gyri and lateral premotor cortex (Binder et al., 2005).

This variability of results may be due, in part, to the fact that in all earlier studies, widely defined categories of abstract and concrete words were compared with each other. Since the brain basis of concrete action and object meaning can vary substantially, it is reasonable to assume similar local specificity for abstract concepts as well. Crucially, the cognitive abstraction processes necessary for understanding grammatical function words, such as “it” or “nevertheless”, are fundamentally different from those related to emotionally related words, such as “hope”, which are, in turn, different from those triggered by words referring to visual shapes, such as “rhomb”. Thus, the cognitive processes of abstraction differ between subtypes of abstract words, and so may the loci of their cortical processing. It may therefore be appropriate for obtaining insights into processes of abstraction to focus on subtypes of words that exhibit specific features of abstractness.

Here, we investigated items that still have some links to action and object features, but abstract away from concrete entities. We looked at words semantically related to color and form, whose meaning is at an intermediate level on the continuum between highly concrete imageable object-related words and highly abstract words. According to the MRC Psycholinguistic Database (Coltheart, 1981; Wilson, 1988), concreteness ratings for highly abstract words are above 600 (examples: “mouse” - 624; “eye” - 634), highly abstract words end are rated below 400 (“love” – 311; “plan” - 357), and our stimuli were in the intermediate range (between 400 and 600; blonde – 502; curve – 447). As our stimuli were carefully matched for important properties, including physical features, visual imageability, familiarity, abstractness and a range of psycholinguistic variables, a unitary semantic system approach would predict congruent activation of the unitary semantic system for both word categories. A version of the sensorimotor theory, however, may predict differential activation of frontal and temporal circuits as a function of the action- and object-relatedness of the semantics of the stimulus words.

Whereas a concrete word, such as “door” directly relates to its reference object, a form or color word, such as “rectangle”, refers to a feature of most of its reference objects that needs to be separated and extracted from other object features. This visual feature extraction process can be computed by neurons in the vicinity of the areas engaged in object processing. Elementary color features of visual stimuli are processed already in primary visual cortex

5

(Hubel, 1995) and temporo-occipital areas have been found to preferentially respond to color processing (McKeefry and Zeki, 1997). To compute a color concept – which covers a range of possible colors, with different levels of brightness and saturation – several alternative feature conjunctions must be classified together, possibly by neural units at higher levels located at gradually more anterior cortical sites. Neural correlates of color concept processing may therefore be present in the temporal lobe, anterior to the temporo-occipital and fusiform areas preferentially activated during color perception (Martin et al., 1995). In animal research, neurons in anterior peri- and entorhinal areas have been proposed to compute increasingly complex conjunctions and disjunctions of features represented by neurons in inferior temporal lobe and closer to the visual input (Bussey and Saksida, 2002; Bussey et al., 2002; Tyler et al., 2004). In the same way, concrete objects can be classified as exhibiting the same shape (e.g., rhomb shape). Establishing a form/shape concept necessitates computation of disjunctions over large sets of concrete objects and elementary visual feature. Again, the neural substrate of progressively abstract computations may be in primary visual (elementary visual features, Hubel, 1995), lateral occipital (object shapes, Kourtzi and Kanwisher, 2001) and inferior and medial temporal cortex. However, in contrast to color concepts, shape concepts always have a motor correspondence, since shapes, but not colors, can be outlined or adumbrated by specific sequences of body movements. Also, when looking at a shape, eye movements follow the shape contour. Because the form of objects can be outlined with different body parts, the brain basis of abstract form concepts may comprise sensorimotor, especially premotor cortex, where action representations are still somatotopic (Rizzolatti et al., 2001), but also neurons in prefrontal cortex related to motor planning (Fuster et al., 2000). Prefrontal cortex just anterior to premotor cortex would be ideally placed for computing conjunctions and disjunctions over sets of specific action programs and could therefore provide a neural substrate for action aspects of abstract form.

Here, we set out to investigate the brain basis of words related to abstract color and form concepts (examples: “brown”, “blonde” and “bronze” vs. “rhomb”, “square”, “arc”). Word groups matched for physical features, visual imageability, familiarity, abstractness and a range of psycholinguistic variables were presented in a passive reading task while event-related blood-flow changes were measured. We hypothesized that the color and form related meaning of these words is reflected by category-specific activation in temporal lobe, and that form-related words specifically activate premotor and prefrontal cortex.

6

3. Experimental procedures

3.1 Imaging methods

Fourteen monolingual, right-handed, healthy native English speakers participated in the study. Their mean age was 25 years (S.D. 5). Subjects were scanned in a 3T Bruker MR system using a head coil. Echo planar imaging (EPI) sequence parameters were TR = 3.02 s, TE = 115 ms, flip angle = 90 degrees. The functional images consisted of 21 slices covering the whole brain (slice thickness 4mm, inter-slice distance 1mm, in plane resolution 1.6*1.6 mm, FOV: 20cm, matrix size 21*128*128). Imaging data were processed using SPM99 software (Wellcome Department of Cognitive Neurology, London, UK).

Images were corrected for slice timing, and then realigned to the first image using sinc interpolation. Phasemaps were used to correct for inaccuracies resulting from inhomogeneities in the magnetic field (Cusack et al., 2003; Jezzard and Balaban, 1995). Any non-brain parts were removed from the T1-weighted structural images using a surface model approach (“skull-stripping”) (Smith, 2002). The EPI images were co-registered to these skull-stripped structural T1-images using a mutual information co-registration procedure (Maes et al., 1997). The structural MRI was normalized to the 152-subject T1 template of the Montreal Neurological Institute (MNI). The resulting transformation parameters were applied to the co-registered EPI images. During the spatial normalization process, images were resampled with a spatial resolution of 2*2*2mm3. Finally, all normalized images were spatially smoothed with a 12 mm full-width half-maximum Gaussian kernel, globally normalized, and single-subject statistical contrasts were computed using the general linear model (Friston et al., 1998). Low-frequency noise was removed with a high-pass filter with time constant 60s. Group data were analyzed with a random-effects analysis. A brain locus was considered to be activated in a particular condition if 20 or more adjacent voxels all passed the threshold of p = 0.001 (uncorrected). In some cases, correction for multiple comparisons in the left hemisphere was administered using the False Discovery Rate correction approach (p < 0.05) (Genovese et al., 2002). Stereotaxic coordinates for voxels with maximal z-values within activation clusters are reported in Talairach space (Talairach and Tournoux, 1988).

Clusters in the left language-dominant hemisphere, which were found to be active in the random-effects analysis of the contrast words vs. matched hash mark stings (Figure 1), were used to define three “canonical” regions of interest (ROIs), in inferior frontal (IF), fusiform (FUS) and parahippocampal gyrus (PH), respectively (see Table 1). Additional “supplementary” ROIs were defined on the basis of earlier findings. Since previous research found an area in the middle temporal gyrus whose activation related to semantic and lexical processing (Chao et al., 1999; Devlin et al., 2004), a middle temporal gyrus (MT) ROI was selected (3*3*3 voxels centered at –60-40/0). Because a large proportion of precentral gyrus was activated by action words in an earlier study (Hauk et al., 2004) and earlier work indicated involvement of premotor cortex in abstract word processing (Binder et al., 2005), possible involvement of premotor cortex dorsal to the inferior frontal language area was probed (–45/0/35). As abstract action concept processing was predicted in dorsolateral prefrontal cortex and earlier work confirmed involvement of the middle frontal gyrus in

7

abstract concept processing (Binder et al., 2005), an additional ROI was placed in a dorsolateral prefrontal area and anterior to premotor cortex (–50/25/25). Previous research has documented activation of this region in a range of tasks tapping into typical prefrontal functions, as for example, abstract mappings between stimuli and responses (Duncan and Owen, 2000). For each subject and each of the 6 ROIs, average parameter estimates over voxels were calculated. These values were subjected to Analyses of Variance including the factors Region of Interest and Word Category (color and form). An additional ANOVA was carried on separately on data from canonical ROIs only. T-tests were used for planned comparison tests of between condition differences in individual ROIs. The mean values scaled to HRF-peak percentage signal change relative to the mean over all voxels and scans within the corresponding ROI and standard errors over subjects are shown in Figure 2.

3.2 Stimuli and experimental design

Words semantically related to visual information, 50 color- and 50 form-related items, were selected using established procedures (Pulvermüller et al., 1999). Stimulus groups were matched for word length counted in letters and syllables (4.3 vs. 4.1 letters, all mono-syllabic), standardized lexical frequency (27.9 vs. 30.9 occurrences/million words, Francis and Kucera, 1982) (also matched according to Baayan et al., 1993), imageability and concreteness/abstractness (mean concreteness ratings (SE): 533 (9.2) vs. 522 (8.6), Coltheart, 1981; Wilson, 1988). Stimulus word groups were also matched for their semantic relation to objects that can be visually perceived, but that they differed in their semantic relatedness to shape and color, as assessed in a rating study (Figure 1, bottom). Subjects were asked whether they considered the word as being semantically related to (i) an object that can be perceived visually, (ii) a visual shape or form and (iii) a color. Ratings were given on a scale from 1 to 7, following procedures described in an earlier publication (Hauk and Pulvermüller, 2004). 150 filler words and 50 pseudo-words were added in order to avoid focussing the subjects’ attention on specific aspects of the stimuli. Stimuli employed during 150 baseline trials consisted of strings of meaningless hash marks varying in length and matched to the word stimuli in length. In addition, 50 null events were included during which a fixation cross remained was displayed. Stimuli were presented for 100ms, and the stimulus onset asynchrony (SOA) between two stimuli was 2.5 s, so that stimulus presentation and scanner trigger were out of phase by ~500ms. Two pseudo-randomized stimulus sequences were alternated between subjects. For statistical analysis, the SPM99 canonical haemodynamic response function (HRF) was used to model the activation time course.

8

4. ResultsWords activated an area in the left fusiform gyrus in the inferior temporal lobe

(activation peaks at -36/-38/-24 and -42/-47/-13) close to and overlapping with the site labeled the visual word form area (VWFA, -42/-57/-15, McCandliss et al., 2003). In addition, an area in inferior frontal cortex comprising the posterior part of Broca’s area, Brodmann’s area 44, and its anterior section, Brodmann area 45, became active (activation peaks at –52/10/20 and –46/28/12). The activation of the VWFA together with inferior frontal areas during passive reading of single words is consistent with earlier findings (e.g., Dehaene et al., 2002; Hauk et al., 2004). There was an additional activation focus in anterior parahippocampal gyrus (-28/-12/-18, Figure 1, Table 1). No significant activation was seen in the right hemisphere.

Figure 1: Cortical activation during passive reading of concrete words semantically related to visual information (p = 0.001, uncorrected). A lateral view is shown on the upper left and

9

01234567

visual color form

semantic feature rated

ratin

g color words

form words

frontal slices of an average brain are displayed on from top left to bottom right. The diagram at the bottom left gives semantic ratings (and their standard errors) for color and form words; ratings of semantic relatedness to general visual information, form information and color information are given (see Experimental Procedures).

Table 1: Areas activated by words relative to strings of hash marks of the same length. Talairach coordinates along with t-values are given for the maximally activated voxel in each local cluster. All clusters that passed the significance threshold at p < 0.001 are listed. The FDR correction cutoff was at T = 3.8.

Brain region Talairachx/y/z

T(13)

All words

Fusiform gyrus, BA 20 LH –36 –38 –18 5.54

Fusiform gyrus LH –42 –47 –13 4.64

Inferior frontal gyrus, BA 45

LH –46 28 10 4.91

Parahippocampal gyrus LH –28 –12 –15 4.69

Comparison of color and form words showed that both word categories activated the inferior temporal VWFA. This activation was more pronounced for form than for color words.

Closer examination indicated that posterior parts of the activated fusiform area was most responsive to color words, whereas anterior fusiform activation was strongest for form words (x=-44, y=-48, z=–20 vs. x=-38, y=-36, z=-18; see Table 2 and Figure 2). The observed differential activation of ventral temporal cortex to color and form words is consistent with a role of this area in the processing of word meaning (Price and Devlin, 2003).

The parahippocampal area was significantly activated by color words, but not by form words. Additional areas in middle temporal gyrus were activated specifically by form words. The most robust and widespread activation elicited by form words was found in left frontal cortex. Significant activation peaks were present in premotor cortex (BA 6, x=-50, y=-2, z=38), inferior prefrontal cortex (BA 45 and 47, x=-34, y=18, z=-2) and dorsolateral prefrontal cortex (BA 9 and 46, x=-54, y=12, z=32 and x=-46, y=26, z=19).

10

Figure 2: Top right: Cortical activation during passive reading of color (in green) and form-related words (red). Center: Frontal slices of the MNI average brain showing regions where specific activation was seen during reading of color (green/blue/purple) or form (red/orange/yellow) words. Bottom left: Effect sizes for color and form words in the six regions of interest in the left hemisphere (rescaled to HRF-peak percentage signal change relative to the mean over all voxels and scans within the corresponding ROI): IF – inferior frontal gyrus; PF- dorsolateral prefrontal cortex; PM – dorsolateral premotor cortex; PH – parahippocampal gyrus; MT – middle temporal gyrus; FUS – fusiform gyrus. The little brain at the bottom right gives the approximate locations of the six regions of interest.

11

-0.5

0

0.5

1

1.5

IF PF PM PH MT FUS

region of interest

% s

igna

l cha

nge

color wordsform words

Table 2: Areas activated by words associated with color and form information. Talairach coordinates along with t-values are given for the maximally activated voxel in each local cluster. All clusters that passed the significance threshold at p < 0.001 are listed. The FDR correction cutoff was at T = 4.3.

Brain region Talairach

x/y/z

T(13)

Color words

Parahippocampal gyrus LH –30 –14 –20 5.60

Cerebellum LH –36 –40 –24 4.38

Fusiform gyrus, BA 20 LH –44 –48 –20 3.98

Form words

Fusiform gyrus, BA 20 LH –38 –36 –18 6.21

Middle temporal gyrus, BA 19/39 LH –38 –63 18 4.88

Middle temporal gyrus, BA 21 LH –57 –43 4 4.28

Inferior frontal gyrus, BA 47 LH –34 17 –2 7.03

Precentral gyrus, BA 6 LH –49 0 35 6.29

Middle frontal gyrus, BA 46 LH –46 26 19 5.85

Middle and inferior frontal gyrus, BA 9 LH –53 13 29 4.91

Precentral gyrus, BA 4 RH 36 –26 55 5.38

Lentiform Nucleus, Putamen LH –20 10 7 4.73

To demonstrate that stimulus types activate specific brain regions to different degrees, it is necessary to directly compare the activity patterns they elicit in pre-defined regions of interest (ROIs). A repeated measures two-way Analysis of Variance was therefore performed to assess

12

significance of word-category-specific activation in the 3 canonical ROIs defined on the basis of the word-hash mark contrast. This revealed a significant interaction of the factors Word Category and Region Of Interest (F (2,26) = 9.11, p < 0.001). Analysis of all 6 regions of interest (fusiform, parahippocampal, middle-temporal, premotor, inferior-frontal and dorsolateral prefrontal areas) confirmed the critical interaction of the factor Word Category with Region Of Interest (F (5,65) = 3.16, p < 0.01). There was also a significant main effect of the factor Word Category (F (1,13) = 6.22, p < 0.02) suggesting stronger activation to form words than to color words. For the 3 canonical regions of interest, planned comparison tests revealed stronger activation to form than color words in inferior frontal and fusiform gyri and the reverse, stronger activation to color than form words, in parahippocampal gyrus (F-values > 4.5, p- values < 0.05). Planned comparison tests performed for the 3 “supplementary” regions of interest revealed significant between-category differences in middle-temporal and prefrontal cortex (F > 4.5, p < 0.05), but not in the premotor region (p = 0.15, n.s.). In summary, these results show that 5 of the 6 Regions of Interest were activated differentially by the word categories under investigation.

13

5. Discussion

Using event-related fMRI while subjects performed a passive reading task, we observed inferior temporal and inferior frontal activation of the left dominant hemisphere to words semantically related to form and color information. Critically, color and form words elicited category-specific cortical differences in focal areas of frontal and temporal cortex. Parahippocampal areas were more strongly activated by color words than by form words, and stronger activity to form words than to color words was seen in fusiform gyrus, middle temporal gyrus, as well as premotor and dorsolateral prefrontal cortex in the inferior and middle frontal gyri. These results provide further evidence that words from different semantic categories – here: color and form words – can activate cortical areas to different degrees, possibly revealing the cortical topographies of their category-specific distributed semantic memory networks or cell assemblies. These findings are in line with previous neuropsychological evidence suggesting that dissociable brain systems contribute to conceptual processing of color and form information (Miceli et al., 2001; Zeki et al., 1999).

5.1 General word related activation

Passive reading of all word stimuli grouped together led to enhanced blood flow in the left temporal lobe, in fusiform cortex, anterior to an area previously related to the processing of prelexical word-related or letter string information (Cohen et al., 2000; Dehaene et al., 2002). Because the stimulus words under study were highly imageable and semantically related to objects known through the visual channel, the present data are also open to the interpretation that word-evoked fusiform activation indicates semantic processing (Price, 2000; Price and Devlin, 2003). In this context, it is noteworthy that the fusiform activation maximum elicited by color words was posterior to that elicited by form words, and that fusiform activation was generally stronger for form words than for color words. This argues in favor of a semantic origin of the present fusiform activation, that is, a specialization of subareas of fusiform gyrus in the conceptual processing of color and/or form information. Apart from temporal cortex, left-inferior frontal cortex, including Broca’s area, was also generally activated by all word forms. This confirms earlier reports on a general role of this brain region in word processing (Binder et al., 1997; Hauk et al., 2004; Price et al., 1996; Pulvermüller et al., 2003; Wilson et al., 2004). The inferior-frontal neurons included in word-related neuronal ensembles would thus be activated not only during speech but during language comprehension as well (Pulvermüller and Preissl, 1991). The role of these inferior frontal neurons may be analogous to that of mirror neurons with a role in action control but also responding to perceptual aspects of actions (Gallese et al., 1996; Iacoboni et al., 1999; Kohler et al., 2002; Rizzolatti et al., 2001), in this case acoustic signals uniquely identifying lexical elements (Fadiga et al., 2002). However, we cannot decide on the basis of the present neuroimaging data, whether the inferior frontal activation is due to mirror neurons or to other

14

so-called canonical multimodal neurons that do not respond specifically to actions, but rather to objects involved in actions (Rizzolatti and Craighero, 2004). The left-hemispheric parahippocampal activation not generally seen for words could tentatively be attributed to the abstract meaning aspects of the stimulus words (but see discussion below).

5.2 Category specific semantic activationCategory-specific activation was reported in a variety of earlier studies (e.g., Dehaene,

1995; Martin et al., 1995; Martin et al., 1996). However, as argued previously (for review, see Pulvermüller, 1999), much earlier work did not control for important physical, psychological and psycholinguistic stimulus features, therefore making it difficult to draw inferences on a semantic origin of the observed effects. Although category-specific brain activation was not always observed when these factors were empirically assessed and controlled for (Devlin et al., 2002), a number of studies confirmed differences between the brain activation patterns elicited by words from different semantic categories. Differences were found between widely defined and increasingly specific categories, ranging from content and function words, nouns and verbs, animal and tool names, to action word subtypes with different meaning (e.g., Brown and Lehmann, 1979; Cappa et al., 1998; Chao et al., 1999; Devlin et al., 2005; Hauk et al., 2004; Kiefer, 2001; Koenig et al., 1998; Preissl et al., 1995; Pulvermüller et al., 1995; Pulvermüller et al., 2005). The present study provided particularly extensive stimulus control and matching of word categories for variables including word length, standardized lexical frequency, familiarity, abstractness, imageability and the degree to which the words were semantically related to visually perceivable objects. On the other hand, empirical evidence was gathered that stimulus groups differed significantly on semantic scales of color- and form-relatedness (Figure 1), a difference at the cognitive level, which was reflected by differential activations of a range of cortical areas. The differential activation to color and form words in different sets of regions of interest argues in favor of an interpretation in terms of semantic and conceptual information linked to the word stimuli at an abstract level.

5.3 Specific activation in temporal lobe

Our results provide evidence that cortical areas differentially contribute to the semantics of color and form words and therefore support established theories of category specific semantic processes in human cortex (Shallice, 1988; Warrington and McCarthy, 1983). Information about color and form is mainly extracted from the visual input and therefore cortical processing differences were expected in the inferior-temporal ventral stream of visual processing. The observed category differences in fusiform gyrus and middle temporal gyrus are consistent with previous imaging work and neuropsychological studies revealing distinct inferior temporal systems preferentially engaging in color and form processing (e.g., Martin and Chao, 2001; Martin et al., 1995; McClelland and Rogers, 2003). An explanation in terms of different inferior temporal neural systems specialized for color and form information can account for the differential activation of fusiform gyrus, in particular the 12 mm distance between form and color word maxima. The specific activation of parahippocampal cortex to color words is consistent with results of recent studies of fine-grained neurospychological deficits in processing color-knowledge related to objects,

15

suggesting that “the mesial temporal stuctures are specifically involved in representing or accessing object color knowledge” (Miceli et al., 2001).

As color and form words differentially activated areas in parahippocampal, fusiform and middle temporal gyrus, it might be possible to relate these differences to aspects of the meaning of these stimuli. Parahippocampal activation specific to color words can tentatively be explained by the visual feature conjunction model (Bussey and Saksida, 2002; Bussey et al., 2002): To classify visual information under a concept, such as a color, logical operations including “and” and “either-or” computations need to be performed by neural elements over a range of input patterns. It is well-known that this requires additional sets of neural elements operating on the input (Kleene, 1956; McClelland and Rumelhart, 1985; Minsky and Papert, 1969), which, in the inferior visual stream must be located progressively anterior to visual cortex. It is possible that feature conjunctions and disjunctions for color concepts, for example binding the different tones and luminances covered by a given color concept, are computed by parahippocampal neurons receiving input from posterior color-related inferior temporal areas. This is consistent with the idea that aspects of color concepts linked to words are processed by anterior temporal sites, thereby supporting the posterior-anterior model of visual feature abstraction (Bussey and Saksida, 2002; Bussey et al., 2002; Tyler et al., 2004). However, as form words did not activate parahippocampal cortex although their meaning, similarly to color words, requires processes that abstract away from any specific object (feature conjunctions and disjunctions), it seems appropriate to postulate that other areas in temporal cortex can contribute to conjunctive processing, too. For form words, the fusiform and middle temporal cortices seem to be relevant here and the suggestion may therefore be that there are different streams in temporal lobe that can play a category-specific role in feature conjunction or disjunction processing.

5.4 Specific activation in frontal lobe

All words under study were strongly related to visually defined perceptual information, but lacked any obvious action meaning, as, for example, tool names and action verbs usually do. Established theories of category-specificity would therefore predict activity differences between color and form words related to visual information processing in temporal lobe, but not in frontal cortex. The specific inferior frontal and middle frontal activation to form words can be tentatively accounted for as follows. An action relationship of form words is evident from the fact that a shape or form can always be outlined or adumbrated by a specific set of body movements. Premotor activation seen for form words may therefore be related to implicit action aspects of the meaning of form words. However, there is a clear difference between the action relatedness of a concrete action word, such as “kick”, and an abstract form word, such as “rhomb”. The former relates to a defined body part whereas the latter refers to a shape that can be adumbrated using the hand or finger, but equally well by a head movement or a series of saccades. The form related action concept should therefore include information that abstracts away the body part information but leaves open the alternatives of using different body parts for perceptually-related actions, which we could expect in prefrontal cortex adjacent to premotor areas.

To directly test this prediction, an earlier experiment on action words related to different body parts (face- and arm- related action words, such as “lick” and “pick” Hauk et

16

al., 2004) was repeated with the participants in the color-/form-word study. Consistent with the earlier findings, face- and arm-words produced somatotopic activation in motor and premotor cortex, which partly overlapped with activation during movements of corresponding body parts. The left inferior-frontal activation to form words overlapped with action word activation (Figure 3, diagram on the left), but no clear overlap was seen between word activation and the premotor activation seen during simple body movements (diagram on the right). Crucially, frontal activation elicited by form words was not restricted to premotor areas (as that seen for concrete action words), but extended into dorsolateral prefrontal cortex in both inferior and middle frontal gyri including Brodmann areas 9 and 46. These areas are anterior to premotor cortex and therefore ideally located for controlling premotor activity. We suggest that the dorso-lateral prefrontal activation specific to form words relates to the activation of neural networks computing conjunctions and disjunctions on concrete action patterns therefore specifying action concepts at an abstract level. The activation of a left inferior area including middle frontal gyrus and premotor cortex is consistent with earlier findings about abstract words in general (Binder et al., 2005). The premotor activation found for form words did not reliably differentiate this word category from color words.

Figure 3: Lateral view of the brain with activation maps for abstract form/shape word (in red) contrasted with the areas activated by concrete action words related to specific body parts (face in blue, arm in green; diagram on the left) and actions performed with these body parts (diagram on the right) (cutoffs: p < 0.001, uncorrected, for words, p < 0.05, FDR-corrected, for actions). Note that form words elicited prefrontal activation in the second frontal gyrus not evoked by concrete action words or actions (yellow circle). All word activations overlapped in the left inferior frontal cortex (white area, diagram on the left) but actions and abstract word processing did not elicit overlapping activation patterns (right diagram).

17

Form wordsTongue movementsFinger movements

Form wordsFace wordsArm words

6. ConclusionOn the basis of these data, the sensorimotor theory of conceptual and semantic

processing (Humphreys and Forde, 2001; Warrington and Shallice, 1984) can be extended to incorporate a model of concepts that may be considered as partially abstract. Neuron populations storing abstract concepts would accordingly be housed in areas adjacent to, and connected with, areas where more elementary and concrete concepts are laid down. Modality specific systems may be located in inferior temporal, middle temporal and parahippocampal gyri that compute feature conjunctions and dysjunctions on visual form and color features. The mirror and/or canonical neuron system in premotor cortex as a site where actions and perceptions are linked together offers a unique basis for action-related abstraction, which could be carried out by neurons in the adjacent prefrontal cortex performing either-or-operations on motor patterns stored in premotor cortex. This view, which covers partially abstract concepts still grounded in action or visual concepts, suggests similar mechanisms of abstraction in different cortical lobes for specific abstract categories. Future research may address the question whether such differential cortical contributions can also be seen for categories of highly abstract lexical items.

In conclusion, we report evidence that two subtypes of partially abstract concepts have their respective brain basis in specific temporal and frontal areas. Color/form word-evoked category-specific activation in fusiform, middle temporal and inferior frontal gyri may be shared in part with other semantic categories. Specific activation in dorsolateral prefrontal and parahippocampal gyri may be most characteristic for abstraction processes necessary for classifying sensory and motor patterns into color and form concepts.

18

7. References

Baayan H, Piepenbrock R, van Rijn H (1993) The CELEX lexical database (CD-Rom). University of Pennsylvania, PA: Linguistic Data Consortium.

Binder JR, Frost JA, Hammeke TA, Cox RW, Rao SM, Prieto T (1997) Human brain language areas identified by functional magnetic resonance imaging. J Neurosci 17: 353-362.

Binder JR, Westbury CF, McKiernan KA, Possing ET, Medler DA (2005) Distinct brain systems for processing concrete and abstract concepts. Journal of Cognitive Neuroscience 17: 905-917.

Brown WS, Lehmann D (1979) Verb and noun meaning of homophone words activate different cortical generators: a topographic study of evoked potential fields. Experimental Brain Research 2: 159-168.

Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Seitz RJ, Zilles K, Rizzolatti G, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. European Journal of Neuroscience 13: 400-404.

Bussey TJ, Saksida LM (2002) The organization of visual object representations: a connectionist model of effects of lesions in perirhinal cortex. Eur J Neurosci 15: 355-364.

Bussey TJ, Saksida LM, Murray EA (2002) Perirhinal cortex resolves feature ambiguity in complex visual discriminations. Eur J Neurosci 15: 365-374.

Cappa SF, Perani D, Schnur T, Tettamanti M, Fazio F (1998) The effects of semantic category and knowledge type on lexical-semantic access: a PET study. Neuroimage 8: 350-359.

Chao LL, Haxby JV, Martin A (1999) Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nature Neuroscience 2: 913-919.

Cohen L, Dehaene S, Naccache L, Lehericy S, Dehaene-Lambertz G, Henaff MA, Michel F (2000) The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain 123 ( Pt 2): 291-307.

Coltheart M (1981) The MRC Psycholinguistic Database. Quarterly Journal of Experimental Physiology 33A: 497-505.

Cusack R, Brett M, Osswald K (2003) An evaluation of the use of magnetic field maps to undistort echo-planar images. Neuroimage 18: 127-142.

Dehaene S (1995) Electrophysiological evidence for category-specific word processing in the normal human brain. Neuroreport 6: 2153-2157.

Dehaene S, Le Clec HG, Poline JB, Le Bihan D, Cohen L (2002) The visual word form area: a prelexical representation of visual words in the fusiform gyrus. Neuroreport 13: 321-325.

Devlin JT, Jamison HL, Matthews PM, Gonnerman LM (2004) Morphology and the internal structure of words. Proc Natl Acad Sci U S A 101: 14984-14988.

Devlin JT, Rushworth MF, Matthews PM (2005) Category-related activation for written words in the posterior fusiform is task specific. Neuropsychologia 43: 69-74.

19

Devlin JT, Russell RP, Davis MH, Price CJ, Moss HE, Fadili MJ, Tyler LK (2002) Is there an anatomical basis for category-specificity? Semantic memory studies in PET and fMRI. Neuropsychologia 40: 54-75.

Duncan J, Owen AM (2000) Common regions of the human frontal lobe recruited by diverse cognitive demands. Trends Neurosci 23: 475-483.

Fadiga L, Craighero L, Buccino G, Rizzolatti G (2002) Speech listening specifically modulates the excitability of tongue muscles: a TMS study. European Journal of Neuroscience 15: 399-402.

Fiebach CJ, Friederici AD (2004) Processing concrete words: fMRI evidence against a specific right-hemisphere involvement. Neuropsychologia 42: 62-70.

Francis WN, Kucera H (1982) Computational analysis of english usage: lexicon and grammar. Boston: Houghton Mifflin.

Freud S (1891) Zur Auffassung der Aphasien. Leipzig, Wien: Franz Deuticke.Friederici AD, Opitz B, von Cramon DY (2000) Segregating semantic and syntactic aspects of

processing in the human brain: an fMRI investigation of different word types. Cereb Cortex 10: 698-705.

Friston KJ, Fletcher P, Josephs O, Holmes A, Rugg MD, Turner R (1998) Event-related fMRI: characterizing differential responses. Neuroimage 7: 30-40.

Fuster JM, Bodner M, Kroger JK (2000) Cross-modal and cross-temporal association in neurons of frontal cortex. Nature 405: 347-351.

Gallese V, Fadiga L, Fogassi L, Rizzolatti G (1996) Action recognition in the premotor cortex. Brain 119: 593-609.

Genovese CR, Lazar NA, Nichols T (2002) Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15: 870-878.

Grossman M, Koenig P, DeVita C, Glosser G, Alsop D, Detre J, Gee J (2002) The neural basis for category-specific knowledge: an fMRI study. Neuroimage 15: 936-948.

Hauk O, Johnsrude I, Pulvermüller F (2004) Somatotopic representation of action words in the motor and premotor cortex. Neuron 41: 301-307.

Hauk O, Pulvermüller F (2004) Neurophysiological distinction of action words in the fronto-central cortex. Human Brain Mapping 21: 191-201.

Hickok G, Poeppel D (2000) Towards a functional neuroanatomy of speech perception. Trends in Cognitive Sciences 4: 131-138.

Hodges JR (2001) Frontotemporal dementia (Pick's disease): clinical features and assessment. Neurology 56: S6-10.

Hubel D (1995) Eye, brain, and vision. New York: Scientific American Library.Humphreys GW, Forde EM (2001) Hierarchies, similarity, and interactivity in object

recognition: "category-specific" neuropsychological deficits. Behavioral and Brain Sciences 24: 453-509.

Iacoboni M, Woods RP, Brass M, Bekkering H, Mazziotta JC, Rizzolatti G (1999) Cortical mechanisms of human imitation. Science 286: 2526-2528.

Jackendoff R (2002) Foundations of language: Brain, meaning, grammar, evolution. Oxford, UK: Oxford University Press.

Jeannerod M, Frak V (1999) Mental imaging of motor activity in humans. Curr Opin Neurobiol 9: 735-739.

Jezzard P, Balaban RS (1995) Correction for geometric distortion in echo planar images from B0 field variations. Magn Reson Med 34: 65-73.

20

Kiefer M (2001) Perceptual and semantic sources of category-specific effects: Event-related potentials during picture and word categorization. Memory and Cognition 29: 100-116.

Kiefer M, Spitzer M (2001) The limits of a distributed account of conceptual knowledge. Trends in Cognitive Sciences 5: 469-471.

Kiehl KA, Liddle PF, Smith AM, Mendrek A, Forster BB, Hare RD (1999) Neural pathways involved in the processing of concrete and abstract words. Hum Brain Mapp 7: 225-233.

Kleene SC (1956) Representation of events in nerve nets and finite automata. In: Automata studies (Shannon CE, McCarthy J, eds.), pp 3-41. Princeton, NJ: Princeton University Press.

Koenig T, Kochi K, Lehmann D (1998) Event-related electric microstates of the brain differ between words with visual and abstract meaning. Electroencephalography and Clinical Neurophysiology 106: 535-546.

Kohler E, Keysers C, Umilta MA, Fogassi L, Gallese V, Rizzolatti G (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297: 846-848.

Kounios J, Holcomb PJ (1994) Concreteness effects in semantic priming: ERP evidence supporting dual-coding theory. Journal of Experimental Psychology: Lerning,Memory and Cognition 20: 804-823.

Kourtzi Z, Kanwisher N (2001) Representation of perceived object shape by the human lateral occipital complex. Science 293: 1506-1509.

Maes F, Collignon A, Vandermeulen D, Marchal G, Suetens P (1997) Multimodality image registration by maximization of mutual information. IEEE Trans Med Imaging 16: 187-198.

Martin A, Chao LL (2001) Semantic memory and the brain: structure and processes. Current Oppinion in Neurobiology 11: 194-201.

Martin A, Haxby JV, Lalonde FM, Wiggs CL, Ungerleider LG (1995) Discrete cortical regions associated with knowledge of color and knowledge of action. Science 270: 102-105.

Martin A, Wiggs CL, Ungerleider LG, Haxby JV (1996) Neural correlates of category-specific knowledge. Nature 379: 649-652.

McCandliss BD, Cohen L, Dehaene S (2003) The visual word form area: expertise for reading in the fusiform gyrus. Trends Cogn Sci 7: 293-299.

McClelland JL, Rogers TT (2003) The parallel distributed processing approach to semantic cognition. Nat Rev Neurosci 4: 310-322.

McClelland JL, Rumelhart DE (1985) Distributed memory and the representation of general and specific information. Journal of Experimental Psychology: General 114: 159-188.

McKeefry DJ, Zeki S (1997) The position and topography of the human colour centre as revealed by functional magnetic resonance imaging. Brain 120 ( Pt 12): 2229-2242.

Miceli G, Fouch E, Capasso R, Shelton JR, Tomaiuolo F, Caramazza A (2001) The dissociation of color from form and function knowledge. Nat Neurosci 4: 662-667.

Minsky M, Papert S (1969) Perceptrons. Cambridge, MA: MIT Press.Neville HJ, Mills DL, Lawson DS (1992) Fractionating language: different neural subsystems

with different sensitive periods. Cerebral Cortex 2: 244-258.Noppeney U, Price CJ (2004) Retrieval of abstract semantics. Neuroimage 22: 164-170.O'Craven KM, Kanwisher N (2000) Mental imagery of faces and places activates

corresponding stiimulus-specific brain regions. J Cogn Neurosci 12: 1013-1023.

21

Perani D, Cappa SF, Schnur T, Tettamanti M, Collina S, Rosa MM, Fazio F (1999) The neural correlates of verb and noun processing. A PET study. Brain 122: 2337-2344.

Posner MI, Pavese A (1998) Anatomy of word and sentence meaning. Proc Natl Acad Sci U S A 95: 899-905.

Preissl H, Pulvermüller F, Lutzenberger W, Birbaumer N (1995) Evoked potentials distinguish nouns from verbs. Neuroscience Letters 197: 81-83.

Price CJ (2000) The anatomy of language: contributions from functional neuroimaging. Journal of Anatomy 197 Pt 3: 335-359.

Price CJ, Devlin JT (2003) The myth of the visual word form area. Neuroimage 19: 473-481.Price CJ, Warburton EA, Moore CJ, Frackowiak RS, Friston KJ (2001) Dynamic diaschisis:

anatomically remote and context-sensitive human brain lesions. Journal of Cognitive Neuroscience 13: 419-429.

Price CJ, Wise RJS, Frackowiak RSJ (1996) Demonstrating the implicit processing of visually presented words and pseudowords. Cerebral Cortex 6: 62-70.

Pulvermüller F (1999) Words in the brain's language. Behavioral and Brain Sciences 22: 253-336.

Pulvermüller F (2001) Brain reflections of words and their meaning. Trends in Cognitive Sciences 5: 517-524.

Pulvermüller F, Lutzenberger W, Birbaumer N (1995) Electrocortical distinction of vocabulary types. Electroencephalography and Clinical Neurophysiology 94: 357-370.

Pulvermüller F, Lutzenberger W, Preissl H (1999) Nouns and verbs in the intact brain: evidence from event-related potentials and high-frequency cortical responses. Cerebral Cortex 9: 498-508.

Pulvermüller F, Preissl H (1991) A cell assembly model of language. Network: Computation in Neural Systems 2: 455-468.

Pulvermüller F, Shtyrov Y, Ilmoniemi RJ (2003) Spatio-temporal patterns of neural language processing: an MEG study using Minimum-Norm Current Estimates. Neuroimage 20: 1020-1025.

Pulvermüller F, Shtyrov Y, Ilmoniemi RJ (2005) Brain signatures of meaning access in action word recognition. Journal of Cognitive Neuroscience 17: 884-892.

Rizzolatti G, Craighero L (2004) The Mirror-Neuron System. Annu Rev Neurosci 27: 169-192.

Rizzolatti G, Fogassi L, Gallese V (2001) Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Review Neuroscience 2: 661-670.

Rogers TT, Lambon Ralph MA, Garrard P, Bozeat S, McClelland JL, Hodges JR, Patterson K (2004) Structure and deterioration of semantic memory: a neuropsychological and computational investigation. Psychol Rev 111: 205-235.

Scott SK, Johnsrude IS (2003) The neuroanatomical and functional organization of speech perception. Trends in Neurosciences 26: 100-107.

Shallice T (1988) From neuropsychology to mental structure. New York: Cambridge University Press.

Shtyrov Y, Hauk O, Pulvermüller F (2004) Distributed neuronal networks for encoding category-specific semantic information: the mismatch negativity to action words. European Journal of Neuroscience 19: 1083-1092.

Skosnik PD, Mirza F, Gitelman DR, Parrish TB, Mesulam MM, Reber PJ (2002) Neural correlates of artificial grammar learning. Neuroimage 17: 1306-1314.

Smith SM (2002) Fast robust automated brain extraction. Hum Brain Mapp 17: 143-155.

22

Talairach J, Tournoux P (1988) Co-planar stereotaxic atlas of the human brain. New York.: Thieme.

Tyler LK, Stamatakis EA, Bright P, Acres K, Abdallah S, Rodd JM, Moss HE (2004) Processing objects at different levels of specificity. J Cogn Neurosci 16: 351-362.

Warrington EK, McCarthy RA (1983) Category specific access dysphasia. Brain 106: 859-878.

Warrington EK, Shallice T (1984) Category specific semantic impairments. Brain 107: 829-854.

Wernicke C (1874) Der aphasische Symptomencomplex. Eine psychologische Studie auf anatomischer Basis. Breslau: Kohn und Weigert.

Wilson MD (1988) The MRC Psycholinguistic Database: Machine Readable Dictionary, Version 2. Behavioural Research Methods, Instruments and Computers 20: 6-11.

Wilson SM, Saygin AP, Sereno MI, Iacoboni M (2004) Listening to speech activates motor areas involved in speech production. Nat Neurosci 7: 701-702.

Wise RJ, Howard D, Mummery CJ, Fletcher P, Leff A, Buchel C, Scott SK (2000) Noun imageability and the temporal lobes. Neuropsychologia 38: 985-994.

Zeki S, Aglioti S, McKeefry D, Berlucchi G (1999) The neurological basis of conscious color perception in a blind patient. Proc Natl Acad Sci U S A 96: 14124-14129.

23