Beyond Salience - Interpretation of Personal

8/2/2019 Beyond Salience - Interpretation of Personal

1/22

Beyond salience: Interpretation of personaland demonstrative pronounsq

Sarah Brown-Schmidt a,*, Donna K. Byron b, Michael K. Tanenhaus a

a Department of Brain and Cognitive Sciences, Meliora Hall, University of Rochester, Rochester, NY 14627, USAb Department of Computer Science and Engineering, Ohio State University, USA

Received 30 August 2004; revision received 16 March 2005Available online 25 April 2005

Abstract

Three experiments examined the hypothesis that it preferentially refers to the most salient entity in a discourse,whereas that preferentially refers to a conceptual composite. In Experiment 1, eye movements were monitored as par-ticipants followed spoken instructions such as, Put the cup on the saucer. Now put it/that. . .. The preferred referent wasthe theme (cup) for it and the composite for that (cup on the saucer) with the goal (saucer) rarely chosen. Experiment 2demonstrated that stressing it reduces the number of theme interpretations. Experiment 3 replicated the main findingsfrom Experiment 1, regardless of whether or not the theme was the backward-looking center. The authors conclude thatentities without linguistic antecedents are sometimes preferred over entities with linguistic antecedents and a single con-struct such as salience is insufficient to account for differences among referential forms. Candidate reference-specific

constructs include the availability of conceptual composites and syntactic role. 2005 Elsevier Inc. All rights reserved.

Keywords: Anaphora; Salience; Cognitive status; Demonstrative; Conceptual composite; Eye-tracking; Pronoun

Introduction

A speaker has numerous choices available for refer-ring to an entity that has previously been mentionedor is otherwise salient in the discourse. For example, aglass of whisky can be called it, this, or the drink. A cen-

tral challenge for models of language processing is tounderstand what conditions underlie the choices madeby speakers and how those choices influence referenceresolution for the listener.

One influential approach within linguistics and com-putational linguistics directly relates the referentialforms to the salience of the speaker s representation ofthe referent (Ariel, 1990; Givon, 1983; Gundel, Hedberg,& Zacharski, 1993). Perhaps the most complete salience-based model is the Givenness Hierarchy proposed by

Gundel et al. (1993), as seen in Table 1.The Givenness Hierarchy proposes that entities in a

discourse differ in salience (referred to as cognitive sta-tus), and can be organized in a subsumption hierarchyrather than into mutually exclusive categories. Each sal-ience category entails all lower categories (categories tothe right in the table). Thus, focused items are also acti-vated, familiar, uniquely identifiable, and so forth. Un-stressed pronouns (personal pronouns) and zeropronouns may only be used to refer to entities with

Journal of Memory and Language 53 (2005) 292313

www.elsevier.com/locate/jml

Journal ofMemory and

Language

0749-596X/$ - see front matter 2005 Elsevier Inc. All rights reserved.

doi:10.1016/j.jml.2005.03.003

q This work was supported by NIH Grant HD-27206 to M.K.Tanenhaus.* Corresponding author. Fax: +1 585 442 9216.

E-mail address: [email protected] (S. Brown-Schmidt).
mailto:[email protected]:[email protected]


2/22

focusedstatus. In contrast, demonstrative pronouns andaccented personal pronouns may be used to refer to enti-ties with activated status, viz. entities that have beenevoked into short-term memory by some trigger in eitherthe discourse or the conversational setting. Gundel et al.(1993) define focus as follows: the entities in focus at agiven point in the discourse will be that partially orderedsubset of activated entities that are likely to be continuedas topics of subsequent utterances [p. 279], for examplesubjects and direct objects of matrix sentences [p.279]. Gundel et al. (1993) combine the salience hierarchy

with Gricean-based reasoning. They suggest that infor-mativeness dictates using the appropriate referringexpression that is highest on the Giveness hierarchy.Thus, each form is interpreted as referring to a referentof a particular salience level. For example, use of ademonstrative signals a less salient referent because amore salient referent would have been signaled by a per-sonal pronoun.

The Givenness Hierarchy, and, more generally, theassumption that salience or activation is the unifyingdimension that distinguishes among referential forms,is consistent with a range of results in the language pro-cessing literature, including those inspired by the influ-ential centering theory (Gordon, Grosz, & Gilliom,1993; Grosz, Joshi, & Weinstein, 1995; Grosz & Sidner,1986; van Gompel & Majid, 2004; Walker, Joshi, &Prince, 1998). For example, pronouns and full namesare interpreted more quickly when referring to focusedand non-focused entities, respectively (Gordon et al.,1993; Hudson, Tanenhaus, & Dell, 1986). Almor(1999) has demonstrated similar results for pronounsand definite noun phrases. The salience-based approachalso underlies most computational algorithms for refer-ence resolution, from early work (e.g., Grosz, 1977;Winograd, 1972) up to recent models (Baldwin, 1997;

Strube, 1998; Tetreault, 2001). It is important to notethat while most manipulations of salience have focusedon linguistic factors, non-linguistic factors are generallyunderstood to also affect referent choice via salience.According to salience approaches, then, linguistic andnon-linguistic factors combine to establish the salienceof each potential discourse referent, which in turn affectsreference choice.

However, it remains unclear whether a uniformdimension such as salience, activation, or expectancy(Arnold, Wasow, Losongco, & Ginstrom, 2000) is suffi-cient to account for how different referring expressions

are processed. Clearly, grammatical features associated

with pronouns, such as number and gender, influencereference resolution above and beyond salience (Arnold,Eisenband, Brown-Schmidt, & Trueswell, 2000; Sanford& Garrod, 1989). Potentially more problematic for uni-form salience-based accounts is the possibility that dif-ferent referential forms may place different weights onthe factors that influence choice of referential form forthe speaker and the preferred interpretation for the lis-tener. In such a reference-specific framework, themapping between referential forms and referents is med-iated by multiple factors, with different referential forms

being more sensitive to some factors than others. Thus, acomplete theory of referring expressions must specify theset of constraints and relative weights for each type ofanaphor. Perhaps the clearest evidence in support of thishypothesis comes from Kaiser (2003). Using corpusanalyses, questionnaires and on-line processing studies,Kaiser (2003) and Kaiser and Trueswell (in press) dem-onstrate that the interpretation of the Finnish pronounshan (s/he) and tama (this) can not be described by a uni-dimensional representation of salience. Instead, the re-sults indicate that in contexts where the antecedentsare full NPs, tama tends to refer to low-salience refer-ents, and han to subjects, regardless of word order or sal-ience. In other words, one could hypothesize that inthese contexts, han is more sensitive to information atthe syntactic level, and tama more sensitive to informa-tion at the discourse level.

In this article, we explore related hypotheses withinthe reference-specific framework for the comprehensionof the pronouns it and that, which, as we have seen, areadjacent to one another in the Givenness Hierarchy. Thefirst hypothesis we consider is that English demonstra-tive pronouns preferentially refer to conceptually com-plex or composite entities. The second is that personalpronouns are sensitive primarily to the salience of the

potential referent. In the next section, we discuss currentwork on the pronouns it and that, and outline thehypotheses and predictions of the two frameworksregarding the interpretation of it and that.

In contrast to the extensive literature on the process-ing of personal pronouns (see Garnham, 2001), demon-strative pronouns have received relatively little attentionin the psycholinguistics and computational linguisticsliterature, perhaps because they are less common in textcompared to personal pronouns. However, recent anal-yses of conversational speech find that demonstrativesoccur in much higher proportions in spoken language

than in text (Eckert & Strube, 2000), and in some cor-

Table 1The Givenness Hierarchy

Status: In focus > Activated > Familiar > Uniquely identifiable > Referential > Type identifiableForm: It/them/they This/that That (noun)/

This (noun)The (noun) This (noun) (indefinite use) A (noun)

S. Brown-Schmidt et al. / Journal of Memory and Language 53 (2005) 292313 293


3/22

pora they even appear as frequently as third-person per-sonal pronouns (Byron & Allen, 1998).

Several studies have concluded that demonstrativesrefer to less-salient entities than personal pronouns.Schuster (1988) presented informants with short dis-courses (as in example (1)) in which the form of the pro-

noun alone was varied in the second sentence. Herinformants had different preferred interpretations for itand that, as illustrated in the discourse in (1):

(1) a. John thought about becoming a street person.b1. It would hurt his mother and it would make his

father furious.b2. It would hurt his mother and that would make

his father furious.

Use of the personal pronoun it in the second con-junct of sentence (1.b1) maintains the reference estab-

lished by the first it (Johns becoming a street

person). In contrast, use of the demonstrative pronounthat in the alternative sentence (1.b2) changes the inter-pretation to something like Johns mother being hurtwould make his father furious. In a similar study,Borthen, Fretheim, and Gundel (1997) found the samepattern in both English and Norwegian. Consistentwith the Givenness Hierarchy, this alternation in mean-ing can be attributed to the effect of attentional focus:personal pronouns prefer referents that are highly sali-ent, whereas demonstrative pronouns prefer less-salientreferents.

A similar distinction between it and that was alsofound in several studies of naturally occurring discourse.In a corpus of spoken descriptions of apartment layouts,it was used to refer to the room currently being de-scribed, which is taken to be the local focus, whereasthat was used to refer to portions of the apartment out-side the current focus of attention (Linde, 1979). Lindefound that it and that were for the most part in comple-mentary distribution. However, a few cases violated thispattern, leading her to conclude that there is some over-lap in the conditions for choosing between that and it. Ina set of career counseling interviews, Passonneau (1989,1993) isolated two factors that characterize the differ-

ences between the use of personal and demonstrativepronouns. First, demonstratives were used more thanpersonal pronouns when either the pronoun or its ante-cedent was not the subject of its clause. Second, whenthe speaker made reference to an entity described previ-ously by a clausal or sentential argument, the moreclause-like the antecedent, the more likely it is for aspeaker to refer to it using a demonstrative pronoun.Passonneau concluded that each of these conditionsindicates that demonstratives are used to refer to entitiesof lower salience than those referred to by personal pro-nouns. These observations are also generally consistent

with Webbers (1991) account of discourse deixis, which

characterizes which portions of a text are suitable forreference with a demonstrative pronoun.

However, observations by Channon (1980), suggestthat salience might not be the primary factor determin-ing the choice between it and that. Channon (1980)noted that demonstrative pronouns have looser agree-

ment features than personal pronouns. Demonstrativescan be used to accomplish identity-of-sense reference(anaphoric mention of an object of the same type asthe antecedent, but not the same instance as the anteced-ent) in addition to identity-of-referent anaphora,1 andtheir numeric agreement allows non-singular objects tobe picked out by a singular demonstrative. He suggestedthat speakers choose to use a demonstrative when theantecedent is a composite entity with conflicting or un-clear semantic features, as in (2):

(2) Patron #1: Ill have the hamburger and fries.

Patron #2: Ill have that, too.

Here, the composite of hamburger and fries is com-plex, composed of two individual entities, one of whichis singular, and the other plural. Demonstrative pro-nouns often do not have clearly identifiable linguisticantecedents. As a result, computational algorithms thatwork well for assigning interpretations to personal pro-nouns perform poorly with demonstratives (Byron,2002). It also complicates the determination of saliencefor the possible referents, when the salience calculationdepends on linguistic criteria. For instance, in (2) the

hamburger and fries are mentioned in a coordinationphrase, so each item on its own should be of roughlyequal salience. However, it is unclear how the conjoinedreferent hamburger + fries should be ranked for saliencewith respect to the individual entities (see Kaup, Kelter,& Habel, 2002; Sanford & Moxey, 1995; Sanford, Sturt,Moxey, & Morrow, 2004, for discussion and some ap-proaches to handling plural referring expressions withconjoined referents).

In sum, whereas demonstratives have attributes thatmake them difficult to characterize using a purely sal-ience-based account, the hypothesis that demonstrativesprefer complex entities could provide a more accurate

characterization of the use of these pronouns. The com-plex entity hypothesis accounts for Channons observa-tions as well as those results that have been used toargue that demonstratives are preferentially used to refer

1 One of the reviewers raised the interesting hypothesis thatthat may be more acceptable for identity of sense referencecompared to it, even when morphosyntactic features of thereferent match both pronouns. While this hypothesis warrantsfurther research, the experiments presented here use real world

objects, which makes identity of sense interpretations unlikely.

294 S. Brown-Schmidt et al. / Journal of Memory and Language 53 (2005) 292313


4/22

to entities that are linguistically less salient than thosereferred to by personal pronouns.

In the following experiments, we examine the inter-pretation of it and that to evaluate predictions derivedfrom the salience and reference-specific frameworks.The salience hypothesis, as exemplified by the Givenness

Hierarchy, predicts that the different referential formsconventionally signal discourse referents that differ insalience. As applied to personal and to demonstrativepronouns, the salience approach predicts that addresseesshould interpret personal pronouns as referring to themost salient entities in a discourse, and should typicallyinterpret demonstratives as referring to less salient enti-ties. Because the Givenness Hierarchy claims that it isacceptable to refer to a more salient entity using a lessspecific term, but not vice versa, that could refer to theentity in focus, but unstressed it should never refer toa non-focused entity. However, personal pronouns

which are accented or carry extrastress

can refer to less

salient entities (Akmajian & Jackendoff, 1970; Cahn,1995; Kameyama, 1999; Lakoff, 1971; Nakatani, 1997).Thus, stressed personal pronouns should behave morelike demonstratives.

In contrast, we examine the hypothesis that it prefersto refer to salient entities, and that prefers to refer toconceptually complex entities. The complex entityhypothesis predicts that the demonstrative that is oftenused to refer to complex or composite entities, such asa hamburger and fries, or a cup on a saucer. While theGivenness Hierarchy may accurately describe overallinterpretation patterns, alternations in referring formsare not exclusively a function of the relative accessibilityof referents. Instead, the observed accessibility-formmapping is a by-product of the conditions governinguse of a particular referential expression, with each ref-erential form placing different weights on the factorsthat influence the use and interpretation of referentialexpressions. For example, full noun phrases may be pre-ferred when the speaker chooses to shift focus from oneentity to another, call attention to a particular feature ofan established discourse referent, or introduce a new dis-course topic, a move that may also introduce new dis-course referents (Vonk, Hustinx, & Simons, 1992).

Speakers tendencies to use demonstratives to referto conceptual composites should create distributionalpatterns that result in bias for readers and listeners tointerpret demonstratives as referring to complex refer-ents, when they are available. In contrast, personalpronouns like it are primarily sensitive to salience fac-tors and preferentially used to refer to focused entities.Thus, addressees should interpret it as referring to theentity in focus.

To evaluate these hypotheses, we created short dis-courses in which participants were instructed to moveeveryday objects, such as cups and saucers, or childrens

blocks. For example, the first utterance might instruct

the participant to Put the cup on the saucer. In an utter-ance like this, the theme (the cup) is predicted to be moresalient than the goal (the saucer) due to the fact that thecup is introduced first in the sentence, and also due tothe fact that an item in the grammatical role of directobject is typically considered to be more salient than

items in adjunct positions. Using grammatical roles todetermine the relative salience of entities in a sentencehas a long-standing tradition in computational linguis-tics (see for instance Grosz et al., 1995; Winograd,1972) as well as in theoretical accounts of accessibility(for example Ariel, 1990; Gundel et al., 1993). Theordering typically used for English is Subject > DirectObject > Indirect Object > Adjuncts. This instructionalso creates a potential composite, the cup on the saucer.Crucially, the linguistic expression that introduces thecup on the saucer is not a linguistic constituent in theutterance, whereas the individual entities the cup and

the saucer are. The next instruction contains the per-sonal pronoun it, as in Now put it over by the lamp, orthe demonstrative pronoun that as in Now put that overby the lamp.

In Experiment 1, we manipulated the availability ofconceptual composites in two ways. First, we in-structed the participant to put the theme object (thecup) on the goal object (the saucer) or next to the goalobject. A cup on a saucer intuitively creates a morenatural composite object than a cup next to a saucer.Additionally, reading studies suggest that plural pro-nouns more easily refer to two individuals when theyare closer in space (Carreiras, 1997). Second, we useddifferent categories of objects, either wooden blocksor everyday objects such as cups and saucers. Usingtwo types of objects allows us to test interpretationpreferences for different types of referents. Addition-ally, we expected that two functionally related objectssuch as a cup on a saucer might form a more naturalcomposite than two stacked blocks, which are less indi-viduated from the other objects in the scene. In Exper-iments 2a and 2b, we investigated the role of prosodyin the interpretation of it and that, examining the ef-fects of prosodic emphasis when composites are orare not available in the scene. These experiments test

the hypothesis that accented pronouns refer to non-fo-cused entities (Cahn, 1995; Kameyama, 1999; Naka-tani, 1997). In Experiment 3, we manipulated thecoherence of the discourse to test the hypothesis thatthe preference to interpret it as the most salient entityis stronger when the most salient entity is morestrongly in focus.

Finally, these experiments allow us to investigate anassumption that is often implicit in models of referenceresolution. Most formal theories of reference assumethat all referential expressions, with the exception of cer-tain types of ellipsis and, in some grammatical frame-

works, empty categories, are linked to conceptual or



5/22

discourse entities that are introduced by, or evoked bylinguistic expressions (antecedents) or relevant non-lin-guistic context (e.g., Heim, 1983; Kamp, 1981; Webber,1979). However, many computational and psycholin-guistic models of reference resolution implicitly orexplicitly assume that discourse entities with linguistic

antecedents are given initial priority. Experimentalinvestigations often only examine linguistically intro-duced expressions, and often assume that a referringexpression accesses its discourse referent by reactivat-ing the linguistic antecedent with which it co-refers (e.g.,Nicol & Swinney, 1989). We should note, though, thatcomputational algorithms for reference resolution ini-tially evaluate a ranked list of entities introduced linguis-tically as a heuristic because few systems have a way toevaluate conceptual factors. The underlying theories typ-ically assume that some form of cognitive status, such assalience, determines the set and ranking of entities. The

conceptual composites created in the discourses we exam-ine offer an interesting case because the first instruction ofeach discourseintroduces a focused entity (the theme) anda salient, but non-focused entity (the goal), each of whichhas a linguistic antecedent. In contrast, the conceptualcomposite does not have a linguistic antecedent. Thus, ifentities with linguistic antecedents are evaluated first,thenthe theme and the goal should be evaluated before a com-posite. And, if entities with linguistic antecedents are al-ways more salient that entities without linguisticantecedents, then it should prefer goal interpretationsover composite interpretations.

Experiment 1: Object coherence

This experiment uses the action-based version of thevisual-world paradigm (Tanenhaus, Spivey-Knowlton,Eberhard, & Sedivy, 1995) to compare the predictionsthat the salience and reference-specific frameworks makefor the interpretation of it and that. Both frameworksmake the prediction that it preferentially refers to the fo-cused, or most salient, entity. For the demonstrative, thesalience-based framework predicts that that should beinterpreted as referring to a non-focused entity, whereas

the reference-specific framework makes the predictionthat the demonstrative that is preferentially interpretedas referring to a complex entity such as a conceptualcomposite, when one is available (Channon, 1980). Wemeasured eye movements because the participants off-line choices might not reflect early processing of the pro-

nouns. Initial interpretation preferences should be re-flected in eye movements beginning 200 ms after theonset of the pronoun, based on earlier work on theon-line interpretation of pronouns using the visualworld eye-tracking technique (Arnold et al., 2000). Thisversion of the visual world paradigm uses real-world ref-erents, a feature that allows us to examine the partici-pants final interpretation of the pronoun, and tocompare the status of linguistically and non-linguisti-cally introduced referents.

Method

Participants

Sixteen participants were recruited from the Univer-sity of Rochester undergraduate community and paidfor their participation. All participants were nativespeakers of North American English, with normal orcorrected-to-normal hearing and vision.

Materials

On each trial, four objects were arranged on a tablein front of the participant as illustrated in Fig. 1. Twoclasses of objects were used: childrens blocks and every-day objects.

The childrens blocks were six small, brightly coloredblocks with different shapes (all cuboids) and colors (yel-low, red, purple, blue, orange, and green). The everydayobjects (see first table of Appendix A) were slightly lar-ger than the blocks, and varied more in size and shape.The four objects, such as a cup and a saucer, and a toylamp and a table, were selected to form two (separate)coherent wholes when the objects were placed together.

Experimental trials. Example sets of instructions fromtrials using the childrens blocks and the objects are pre-sented in examples 3 and 4, respectively.

Fig. 1. A 2-D representation of the experimental display, for the blocks (A), and objects (B) conditions of Experiment 1.



6/22

(3) a. Put the red block next to the blue block.b. Now put that on the green block.c. Put the green block on the yellow block.d. Now put the blue block on the yellow block.

(Display includes: a red, blue, green, and yellowblock.)

(4) a. Put the cup on the saucer.b. Now put it over by the lamp.c. Put the table next to the lamp.d. Now put the cup in front of the table.

(Display includes: cup, saucer, table, and lamp.)

In both examples, the first sentence (3a and 4a) intro-duces two discourse entities, one in the role of theme(the object being moved) and one in the role of goal(the destination of the moved object). The theme, whichis underlined above, is predicted to be the most saliententity at the onset of Now in the (b) sentences. The (b)

sentences contained a pronoun, either it or that. Eyemovement data were analyzed from the onset of the pro-noun and continued until participants completed an ac-tion in response to that command. The third and fourthinstructions never contained pronouns, and eye-trackingdata associated with these sentences were not analyzed.Sixty-four instruction sets, each containing four instruc-tions, were used in the study. Thirty-two containedblocks and 32 contained common objects. Half of thesetrials contained a pronoun in the second instruction.

Stimulus sentences were recorded by the first author,speaking as naturally as possible. The natural stress on

that was somewhat more pronounced than the stress onit; that was longer than it by an average of 88 ms. Previousobservations that prosodic differences in a single word canbe foreshadowed in length differences earlier in a phrase(see Arnold, Fagnano, & Tanenhaus, 2003) prompted usto examine length differences in the words preceding thepronoun. Our carrier phrases were Now put it/that nextto. . .. Now was significantly shorter for phrases contain-ing that compared to phrases containing it. While thelength of put was equivalent for that and it phrases, thetime in between the offset ofput and the onset of the pro-noun waslongerfor that phrases, suggesting that thelong-er duration of that is reflected in the prosody of the

preceding words and pauses. We will address these timingissues when we turn to analyses of the eye movement data.

Filler trials. The 32 filler trials and the two filler sen-tences on each target trial (e.g., 3c and d; 4c and d) nevercontained pronouns and were designed to equate theprobability of manipulating each object or composite,given the previous instruction. The sentences in (5) arean example filler trial.

(5) a. Put the nest next to the hamburger.b. Now put the plate under the hamburger.

c. Put the bird in the nest.

d. Now put the hamburger and the bird in front ofthe nest.(Display includes: hamburger, plate, bird, andnest.)

Equating the probability of manipulating each ob-

ject, given the previous instruction, was important in or-der to decrease the likelihood that participants wouldinterpret a critical pronoun as referring to simply themost frequently moved entity (e.g., the theme, goal, orcomposite from the preceding trial).

We manipulated the following three factors: pronountype (it vs. that); object type (everyday objects vs. chil-drens blocks); and whether the prepositional phraseintroducing the goal in the first instruction was on topofor next to. In all conditions, the item that is the themeof the (a) sentence is the predicted focus when the pro-noun is encountered. We will refer to the manipulation

of the preposition as the location manipulation, sincethe preposition determined the location of the themewith respect to the goal. The object type manipulationwas blocked (1st vs. 2nd half of the experiment) andblock order was counterbalanced across participants.Pronoun type and preposition type were both manipu-lated using a modified Latin square design.2 Each partic-ipant was presented with a total of 32 target pronounsentences; four in each of the eight different conditions.The trials using childrens blocks were organized intoone pseudo-random order. The six different coloredblocks were randomly assigned to the sentences fortwo different lists (plus two corresponding reverse-orderlists). The trials using everyday objects were organizedinto a different pseudo-random order, and the individualitems (such as the trial with the cup, saucer, lamp, andtable) were rotated through the four conditions, yieldingfour lists (plus four reverse-lists).

Procedure

After the eye-tracker was calibrated, participantswere seated at a table on which the experimenter placedthe experimental stimuli, which included everyday ob-

jects and childrens blocks. Participants were instructedto follow pre-recorded spoken instructions to manipu-

late the objects. They were told that the task was fairlyeasy and they should do the first thing that comes tomind when they heard the instructions. Eye movementswere measured using a light-weight, head-mounted ASL

2 An error in list design created an imbalance in one of theblocks lists such that one condition appeared 5, rather thanfour times (and another condition occurred 3, rather than fourtimes). In the objects lists, each item was accidentally rotatedthrough three of the four conditions, however the rotationswere balanced such that each person saw an equal number ofitems in each condition. Other than these minor deviations, the

design followed the Latin squares method.



7/22

Series 5000 eye-tracker. Software superimposed fixationson a video-record taken from a 60 Hz camera mountedon the headband. The actions and eye movements of theparticipants were hand-coded by the experimentersusing a frame-accurate digital VCR.

Results

Referent choice

On the experimental trials, e.g., Put the cup next tothe saucer. Now put it/ that. . ., participants typicallymoved either the theme (the cup) or the composite (boththe cup and saucer) to the specified location. We will re-fer to the object(s) they moved as the referent of thepronoun. A small number of responses fell into one ofthree other categories (accounting for 2% of the data):selection of the goal (e.g., the saucer), one of the othertwo items in the scene (e.g., the lamp or table, in Exam-ple 4), or selecting three or more items. Due to the lowfrequency of occurrence, we will not discuss those re-sponses except to note that the composite, which wasnot introduced in a linguistic constituent, was preferredto the goal, which did have a linguistically introducedantecedent.

It and that clearly had different preferred referents.Participants tended to interpret it as the theme of thepreceding utterance and that as the composite (see Figs.2A and B).

However, the availability of a conceptual compositeinfluenced the interpretation of both it and that. Placing

the theme on top of the goal object increased the numberof composite interpretations for both it and that. This ef-fect was strongest for the conditions with everyday ob-

jects. These observations were confirmed by analyses ofvariance, using subjects as the random factor. Items anal-yses for the blocks trials are not informative because theitems in the blocks list were very similar to one another,and the six different colored blocks were randomlygrouped for each trial. An ANOVA by subjects with theproportion of theme responses as the dependent variableand including pronoun type (it/that), object location (ontop/next to), and object type (objects/blocks) as factors

was significant for all three main effects, as well as a signif-

icant object type by object location interaction.3 Theremaining three-way and the two-way interactions werenot significant (all Fs < 1.5). The main effect of pronounwas due to more theme interpretations for it than for that,F(1,15) = 46.23,p < .0001. The main effect of object loca-

tion was due to more theme interpretations in the next tocondition compared to the on top condition, F(1,15) =35.33, p < .0001. The main effect of object type was dueto more theme interpretations in the blocks conditionscompared to the objects conditions, F(1,15) = 6.72,

p < .05. The significant object type by object locationinteraction, F(1,15) = 20.54,p < .001, was due to a great-er preference for the composite interpretation in the ontop/objects condition compared to the on top/blocks con-dition, t (15) = 3.69,p < .01, and no effect of object type inthe next to condition, t (15) = .17, p = .87. A separateitems analysis for the objects condition yielded a patternsimilar to the subjects analysis. Note that a between-itemsanalysis was used due to the problem in list rotation. Theeffects of both pronoun, F2 (1,39) = 49.07,p < .0001, andlocation, F2 (1,39) = 97.22, p < .0001 were significant,and they did not interact.

In summary, the preferred interpretation ofit was thetheme, and the preferred interpretation of that was thecomposite. However, the interpretation of both pro-nouns was modulated by factors that affected how easilythe theme and goal could be construed as a compositeentity. Both pronouns were more likely to be interpretedas the composite when the theme and goal were seman-tically related objects, and when the theme was placed

on top of the goal, making a visually salient grouping.Composite interpretations were strongest in the on top/objects condition where the semantic and visual group-ing cues worked together. In this condition, it and thatwere interpreted as the composite 40 and 88% of thetime, respectively. Despite being linguistically intro-duced in the immediately preceding sentence, the goalwas not a viable referent for interpretation of either pro-noun. The fact that the goal was underneath the theme

Fig. 2. Experiment 1 referent selections for blocks (A) and objects (B) conditions split by object location (on top/next to), and pronoun(it/that). Grey portion, theme responses; white, composite; black, goal.

3 One item from the objects condition was not included in theanalyses because it contained the noun grapes which may not

be suitable for reference with the personal pronoun.



8/22

in the on top conditions may have contributed to this ef-fect, however in the next to conditions, the goal wasphysically as available as the theme, and goal interpreta-tions never exceeded 7% in this condition. Additionally,recall that participants were equally likely to manipulatethe top and bottom objects of a composite in the filler

sentences, which should reduce bias against manipulat-ing the bottom object.

Fixations to potential referents

We analyzed the timing with which participantslooked at the objects in the display in order to examinethe time course with which potential referents were con-sidered. We assume, following other research on anaph-ora using the visual world paradigm, that looks to anobject following an anaphor indicate that the object isbeing considered as a potential referent (Arnold &Eisenband et al., 2000; Runner, Sussman, & Tanenhaus,

2003). We first conducted an analysis of eye movementsbeginning at the onset of the pronoun in phrases like 3band 4b, and continuing until the participant moved andreleased an object in response to the command. Theaverage onsets of it and that were 316 and 358 ms afterthe onset ofNow, respectively. To account for this differ-ence we aligned the eye movement data at the onset ofthe pronoun in each trial separately. The destinationword (e.g., lamp in example 4b) began approximately400 and 500 ms after the onsets of it and that, respec-tively. Fixations to an object were defined as a sequenceof fixations (at minimum two frames) on a single object,or within 1 degree of visual angle of the edge of the ob-

ject. Following the first instruction, the theme and goalitems were physically very close to one another. Thus,distinguishing looks which are close to both the themeand goal is difficult given the resolution limit of theeye-tracker (.1 degree of visual angle, at best). To resolvethis difficulty, we implemented a strict coding scheme.For fixations near two objects, we counted the fixationas being on the object the fixation crosshair was closerto. While limitations of the experimental display andtracker resolution increase noise in the eye-trackingdata, the coding scheme assures that this noise equallyaffects the two critical objects.

If a participant considers the theme to be the referent,we expect increased looks to the theme beginningapproximately 200400 ms following pronoun onsetand decreased looks to the goal. In contrast, if a partic-ipant only looks to the goal, this would suggest that par-ticipants were considering a goal interpretation.However, it is less obvious what the pattern should bewhen the participant considers the composite. In this sit-uation, the participants might fixate the goal object oralternate looking between the theme and goal. Thus,while we expect more variability when participants areconsidering the composite, when comparing goal vs.

composite interpretations, we expect more goal looks

when the participant interprets the pronoun as the goal,and when comparing theme vs. composite interpreta-tions, we expect more theme looks when the participantinterprets the pronoun as the theme.

First, we present analyses of the proportion of eyefixations to the theme and goal across the three condi-

tions that we manipulated: type of object, pronoun,and location. Recall, however, that these analyses areconfounded by the fact that participants made differentnumbers of theme and composite choices across condi-tions. In order to partially account for these differencesin final interpretation, we next present action-contingentanalyses (Runner et al., 2003) in which we separatelyexamine trials for which the participants adopted themeand composite interpretations.

Upon the onset of the pronoun in critical phrases, fix-ations were primarily limited to the theme and goal, witha later rise in fixations to the destination location and

few looks to the other object in the scene. To simplifyour analyses, we focus on the proportion of fixationsto the theme, goal, and destination. Additionally, in or-der to simplify presentation of the eye fixation data, inthe eye fixation graphs (but not the analyses), we col-lapse the data from the two different object types (blocksand objects) together.

Figs. 3A and B show the proportion of fixations tothe theme, goal, and destination for the next to/it, andnext to/that conditions. As with the overall pattern offixations, initial eye movements were primarily to thetheme and goal, followed by a later rise in looks to thedestination location. Eye movements associated withthe pronoun that elicited substantially more looks tothe goal object.

In the next to/it condition, looks to the goal fall be-low 20% at about 1100 ms, in contrast, in the next to/that condition, looks to the goal remain high, reflectingthe larger number of composite interpretations in thiscondition. In the on top/it condition, there was an earlypreference for looks to the theme (Fig. 3C). In contrast,in the on top/that condition (Fig. 3D), we initially see analmost equal proportion of looks to theme and goal, fol-lowed by sustained looks to the goal which do not dropoff as they do in the it condition.

The fixation data were analyzed in three 600 msepochs, beginning 200 ms after the pronoun onset, toallow time for the programming of eye movements(Matin, Shao, & Boff, 1993). The first two epochs cap-ture the earliest effects we would expect to see basedon participants hearing the pronoun. The onset ofthe destination information (e.g., lamp in example4b) begins shortly before the beginning of Epoch 2,so we expect to see destination-related effects primarilyin Epoch 3.

Following Arnold et al. (2003), we analyzed the eye-tracking data in terms of theme advantage, in order to

evaluate the relative preference for the theme over the



9/22

goal. The theme advantage was calculated for each seg-ment and condition separately by subtracting the pro-portion of fixations to the goal from the proportion offixations to the theme. Separate (planned) ANOVAsby subjects at each epoch were conducted to examinethe time-course of eye movements effects. Theme advan-tage was used as the dependent variable, and object type(blocks and objects), location (next to/on top), and pro-noun (it/that) were used as factors. At the first and sec-ond epochs, we observed only a main effect for pronoun,epoch 1: F1 (1,15) = 7.91, p < .05, epoch 2:F1 (1,15) = 17.81, p < .001, due to a higher target advan-tage score (more theme looks) for it compared to that.Thus the pattern of eye movements reflects the partici-pants choices, beginning shortly after the pronoun.

The ANOVA at epoch 3 revealed a main effect ofpronoun type, F1 (1,15) = 21.44, p < .001, due to a lar-

ger theme advantage for it, as well as a marginal effectof object type, F1 (1,15) = 3.71, p = .07. This was dueto a larger theme advantage for everyday objects com-pared to blocks. At the third epoch, we also observeda significant object type by location interaction,F1 (1,15) = 15.17, p < .01, which was due to a largertheme advantage for next to compared to on top in theobjects condition, F1 (1,15) = 7.42, p < .05, and no dif-ference between next to and on top in the blocks condi-tion, F1 (1,15) = .35, p = .56. The items analysis byepoch for the objects condition showed a similar pattern;while no effects were significant at the first epoch, the

pronoun effect was significant at epoch 2, F2 (1, 39) =

8.23, p < .01. At epoch 3, both the pronoun, F2(1,39) =5.61, p < .05, and location effects, F2(1,39) = 5.61,

p < .05, were significant. The interaction did not ap-proach significance.

Taken together, the pattern of looks to potential ref-erents supports two conclusions. First, the strongest andmost consistent effect we observed is a higher proportionof theme fixations for it compared to that; the pronouneffect was significant at all three epochs and for bothblocks and objects. Second, the interpretation of bothpronouns is affected by the availability of a composite,indicated by significant decreases in the theme advan-tage when the objects were on top rather than next toeach other, as well as an increase in composite selectionsfor the on top condition. The theme preference for it isconsistent with the hypothesis that it is primarily usedto refer to the most salient entity. However, the increase

in off-line composite interpretations for it when a com-posite was available and the fact that increasing theavailability of the composite affected the on-line process-ing of it, together suggest that non-linguistically men-tioned entities can sometimes be more salient than thelinguistic focus. Additionally, that showed a compositebias that was also modified by the availability of thecomposite. In fact, in cases where the composite wasavailable, we observed an increase in composite selec-tions for both it and that. This finding is inconsistentwith a purely salience-based account, which would pre-dict that that should be preferentially interpreted as

the less salient entity.

Fig. 3. Experiment 1. Relative proportion of fixations to theme, goal and destination for next to/it (A), next to/that (B), on top/it (C),and on top/that (D), for both blocks and objects.



10/22

The eye movement analyses that we have presentedthus far combine trials in which participants chose thecomposite with trials in which participants chose thetheme. This is standard practice in studies of referenceresolution during reading in which there is no clear mea-sure of the participants final interpretation. However,

this is potentially problematic when comparing ana-phors that have different preferred interpretations, orwhen an anaphor has several potential referents, andthe preferred referent differs across conditions. The taskwe used, however, allows us to bypass this problem byseparating trials based on the participants final interpre-tation of the pronoun. Thus, we conducted action-con-tingent analyses in which we examined eye movementsbased on the final referent choice.

If the eye movement patterns we observed are due toearly, response-independent processing driven solely bythe anaphor, we might expect to see a signature patternof early eye movements for each anaphor, regardless ofthe final interpretation. Alternatively, the earliest eyemovements might reflect the final interpretation.

Action-contingent analyses. Due to problems of smallsample sizes, we collapsed across the object-type vari-able and will focus primarily on descriptive analyses.First, we present the data from trials in the on top/thatcondition in which participants chose the composite(Fig. 4A), because this condition strongly facilitatedcomposite interpretations. Fig. 4B presents fixationsfor trials in the next to/it condition in which participants

Fig. 4. Experiment 1. Relative proportion of fixations to theme, goal, and destination for on top/that (chose composite), and next to/it(chose theme), respectively, for blocks and objects together.

Fig. 5. Experiment 1. Relative proportion of fixations to theme, goal, and destination for both blocks and objects. (A and B) on top/itcondition where participants chose the composite, and the theme (respectively). (C and D) next to/that condition where participants

chose the composite, and theme (respectively).



11/22

chose the theme, because this is the condition that bestsupported the preferred interpretation of it.

The pattern of fixations in Fig. 4A shows strong,long-lasting competition between theme and goal,reflecting the fact that the participants formed compos-ite interpretations on these trials. In contrast, the fixa-

tion pattern in 4B shows an immediate theme bias anda quick drop in the fixations to the goal.

Next, consider the processing ofit in the on top condi-tion when participants selected eitherthe composite or thetheme (Figs. 5A and B, respectively). The fixation patternwhen participantschosethe theme shows thesame patternas the next to/it condition (Fig. 4B). In addition, whenparticipants selected the composite, there was still an ini-tial theme bias, with thecomposite pattern emerging later.These results suggest that listeners initially considered thetheme interpretation for it on all trials, later rejecting thetheme interpretation on some proportion of trials when

there was also a composite available.Figs. 5C and D show the eye movements associated

with the next to/that condition on trials where partici-pants selected the composite and the theme (respec-tively). When participants selected the composite (5C),we see the fixation pattern associated with composites,with looks equally distributed between the theme andgoal. In contrast, when participants chose the theme(5D), we see a substantial theme advantage, comparableto the pattern seen for interpretations of it in the next tocondition (Fig. 4B). In both the next to/that and on top/it conditions, there are two clear patterns of interpreta-tion of the pronoun (as the theme or composite), whichare reflected not only in the behavioral data, but also inthe early on-line data. When it and that were assignedtheme interpretations, the pattern of early eye move-ments was very similar, suggesting that similar referentswere considered. However, for composite interpretationsthere was a suggestion of an initial theme bias for it butnot for that.

Experiment 1: Conclusions

We replicated previous findings by Linde (1979), Pas-sonneau (1989, 1993), Schuster (1988), and Borthen et

al. (1997) that it and that differ in their preferred inter-pretation, with it preferentially referring to the more fo-cused entity. However, we found that it was not onlysensitive to linguistic focus; when a composite was avail-able in the scene (in the objects, on top condition) we ob-served frequent composite interpretations for it. The factthat the action-contingent analyses showed a theme biasfor it regardless of the participants final choice indicatesthat it is highly sensitive to the linguistically determinedfocus even in these cases. However, on the assumptionthat it preferentially refers to the most salient entity, in-creases in composite interpretations in the objects/on top

condition demonstrate that an entity without a linguistic

antecedent can compete with the most salient entity thathas a linguistic antecedent. These results are problematicfor models that assume that entities with linguistic ante-cedents have privileged status during initial referenceresolution.

The results for that are problematic for models in

which reference resolution is based purely on salience.According to a salience account, conditions that increaseit-interpretations should decrease that-interpretationsbecause that does not prefer the most salient alternative,whereas it does. However, in the on top conditions whereit was more likely to be interpreted as the composite,and thus would sometimes be the most salient alterna-tive, there was an increase in composite interpretationsfor that. This is consistent with the prediction that thedemonstrative pronoun would be interpreted as thecomposite if one was available, regardless of its salience.In the next to conditions, where a composite was not as

available, there was no increase in goal fixations or goalinterpretations for that. Rather, fixations to the themeincreased, along with an increase in theme interpreta-tions (compare the fixation patterns for the next to/thatand the on top/that conditions). This result is not entirelyconsistent with the hypothesis that that prefers a less-sa-lient entity, which would have predicted a preference forgoal referents over themes. Instead, we suggest an alter-native explanation that when a complex entity is notavailable, the demonstrative that is interpreted as refer-ring to a task-relevant entity. Here, the theme may bemore relevant to the task than the other objects becauseit was just moved, and the listener could plausibly as-sume that it should be moved again, despite the fact thatfiller trials were designed to decrease the likelihood par-ticipants expected a particular kind of continuation. Thehypothesis that task-relevant objects are preferred forthat when no composite is available will require furtherexploration.

The eye-tracking results for it and that show that ini-tial preferences for the two pronouns diverge starting asearly as 200 ms following the onset of the pronoun, witha larger theme bias for it. This pattern, in conjunctionwith the effects of object location and object type seenin epoch 3 indicate that the eye movement data are

highly consistent with overall off-line selections. More-over, the fact that the eye movement data do not indi-cate a divergence between the overall off-line responsepreferences and on-line interpretations suggests that, inthe following three experiments, we can further exploreparticipants interpretation preferences examining refer-ent selection alone.

Experiments 2a and 2b: Prosodic effects

In Experiment 1, we demonstrated that the interpre-

tation preferences for it and that are clearly different,



12/22

with it preferentially interpreted as the theme and that asthe composite, and that the proportion of compositeinterpretations was modulated by the availability of acomposite. However, an important difference betweenit and that is that that tends to receive more prosodicstress than it. In the stimuli used in Experiment 1, that

was longer than it by an average of 88 ms. The preferredreferent of a stressed pronoun is thought to be less sali-ent than the unstressed counterpart, suggesting thatstress may be a component of the differences betweenthe pronouns observed in Experiment 1. In Experiment2a, we manipulate the availability of a composite andalso vary the stress on it and that. Comparing stressedand unstressed variants of that and it is important forat least three reasons. First, we need to establish thatthe difference in preferred interpretation for it and thatis not simply due to the fact that the pronouns differin stress. Second, on the assumption that stressed pro-

nouns refer to less salient entities, comparing stressedand unstressed versions of it and that should add toour understanding of the relative salience of the dis-course referents. Finally, although there have been anumber of proposals about the interpretation of stressedpronouns, experimental research on the issue has beenequivocal.

Various proposals regarding the use of stress (orpitch accenting) for personal pronouns claim that astressed pronoun is used when the intended referent isnot the focused entity. The classic example (due to Lak-off, 1971) in (6) illustrates the alternation in meaning be-tween unstressed and stressed versions of the subjectpronoun he and object pronoun him. The example in(7) shows a similar alternation (original example dueto Akmajian & Jackendoff, 1970), but does not containthe second pronoun.

(6) a. John called Jim a republican, then he insultedhim.

b. John called Jim a republican, then HE insultedHIM.

(7) a. Lolita slapped Doloris and then she hitHumbert.

b. Lolita slapped Doloris and then SHE hit

Humbert.

By adding stress to the first pronoun in (6), the inter-pretation of he shifts from the focus (John) to the sec-ond-mentioned (thus less accessible) entity, Jim. Noticethat interpretation of the object pronoun also seems tochange (see Smyth, 1994, for a discussion of accentedobject pronouns). Example 7 is more similar to our stim-uli in that the sentence only contains a single pronoun.Here, the unaccented she in 7a is typically interpretedas the focus, Lolita, while the accented she in 7b shiftsthe interpretation to Doloris. Accenting a pronoun is

typically thought to shift the interpretation to a less-sa-

lient entity (see Cahn, 1995; Nakatani, 1997), thoughexamples of stressed pronouns in cases where only a sin-gle referent is available have been used to suggest thatstressed pronouns indicate rhetorical contrast instead(de Hoop, 2004). A different approach by Kameyama(1999) proposes that there is a complementary relation-

ship between stressed and unstressed pronouns in the lo-cal domain. When two or more possible referents are inthat domain, the unstressed pronoun refers to the mostsalient entity and the stressed pronoun refers to the leastsalient entity.

Previous studies examining the effects of prosodicstress on pronoun interpretation have failed to findsystematic effects when prosody is taken as an isolatedcue. Wolters and Byron (2000) analyzed the use ofboth accented demonstratives and personal pronounsin task-related discourse. They did not find any pro-sodic features that could be used reliably to determine

attributes of the pronouns referent, such as the dis-

tance from the antecedent to the pronoun or the syn-tactic properties of the antecedent. In another corpusstudy, stress on subject pronouns with gender wasfound to be a signal of other discourse properties, suchas an implied contrast, rather than a clue to the pro-nouns referent (Wolters & Beaver, 2001). In an on-lineinterpretation study, Venditti, Stone, Nanda, and Tep-per (2002) and Venditti, Trueswell, Stone, and Nautiyal(2003) found that a stressed subject pronoun causesparticipants to look at both potential referents untiladditional material in the sentence helps disambiguatethe pronouns meaning. Therefore, she concluded thatlisteners are unable to use prosodic accent on its ownas a disambiguating clue. We are not aware of any on-line studies comparing stressed and unstressed that.

Experiments 2a and 2b investigated how addingstress to the pronouns it and that affected their interpre-tation. The participants manipulated everyday objectsthat were not easily viewed as composites (e.g., a frogand candle); we will return to this design change in thediscussion. We tested participants using both stressedversions of the pronouns and unstressed versions likethose in Experiment 1. Experiment 2a evaluated threealternative hypotheses: The first is that stressing the per-

sonal pronoun shifts the interpretation away from thefocus. In order to evaluate this hypothesis, we comparethe interpretation of stressed and unstressed versions ofit, with a prediction of fewer theme selections for thestressed pronoun. The second hypothesis is that stress-ing a personal pronoun results in a preference for theleast salient entity, i.e., the goal. The third hypothesisis that the primary difference between unstressed it andthat is stress. In order to evaluate this hypothesis, wecompare unstressed that with both stressed and un-stressed versions of it, with a prediction that unstressedthat should be similar to stressed it but different from

unstressed it.



13/22

Experiment 2b is a follow-up study in which thematerials were recorded differently. The sentences forExperiment 2a were recorded with natural prosody,which resulted in prosodic differences between thestressed and unstressed sentences that emerged beforethe onset of the pronoun. Thus, we cannot be sure

whether the effects of stress on it are due to the extrastress on it, or to a change in the entire prosodic con-tour of the phrase. Experiment 2b used an identical de-sign to Experiment 2a, except for the followingchanges: (1) We carefully spliced the stressed and un-stressed pronouns into the carrier phrases to allowonly the pronoun to change between conditions (onaverage). (2) The word Now was removed from theinstructions because the length of Now was strongly af-fected by the stress on the pronoun. These materialscarefully controlled the prosody, but sounded muchless natural.

Method

The methodology used in Experiments 2a and 2b isgenerally the same as Experiment 1, thus we only notethe differences in the technique. Sixteen participantsfrom the University of Rochester undergraduate com-munity participated in Experiment 2a, and a separateset of 16 participated in Experiment 2b. Each experi-ment lasted approximately 90 min.

Materials

An example instruction set is given in (8). All trialsused the everyday objects for both experiments. Addi-tionally, the objects were not presented in functionallyrelated pairs as they were in Experiment 1. A full listof materials is included in Appendix A.

(8) a. Put the balloon next to the road.b. (Now) put that on the clip.c. Put the hamburger on the balloon.d. (Now) put the road behind the hamburger.

(Display includes: a balloon, road, clip andhamburger. Now was not included in the 2bstimuli.)

In addition to the pronoun (it/that) and preposition(on top/next to) manipulations, we manipulated whetherthe pronoun received extra stress. Each participant saw32 target items, four in each of eight conditions. The tar-get sentences were of the same structure as those inExperiment 1, and were interspersed with 32 fillers ofthe same structure, but which did not contain pronouns.The 32 target items were rotated through the eight con-ditions in eight lists (plus 8 reverse-order lists) using amodified Latin squares design. The items in each listwere randomly ordered, except that adjacent trials were

never in the same condition.

Details of the sentence recordings. Stimulus sentencesfrom Experiments 2a and 2b were pre-recorded by thesame speaker as in Experiment 1. Each critical sentence(b) was separately recorded in four different wayswithit or that, with normal stress or extra stress. The stressedpronouns were the most acoustically prominent word in

the sentence, and were audibly different than the un-stressed versions of the pronouns. We confirmed thestress manipulation by measuring the duration of thestressed and unstressed pronouns, as duration is anacoustic correlate of stress (see discussion in Hirschberg,1993). We also measured the pitch excursion (maximumpitch) of each pronoun, and performed a ToBI analysison a subset of the pronouns from each experiment.

For Experiment 2a, the mean duration of the stressedand unstressed versions of it were 281 ms (standard er-ror = 8.6) and 70 ms (3.9), respectively, and the meanduration of the stressed and unstressed versions of that

were 290 ms (7.1) and 143 ms (6.3), respectively. An AN-OVA showed a main effect of stress, F(1,31) = 600.2,

p < .0001, and a main effect for type of pronoun,F(1,31) = 64.0, p < .0001. A significant stress by pro-noun type interaction, F(1,31) = 31.71, p < .0001 wasdue to shorter durations for it than that in the unstressedconditions, paired t (31) = 12.07, p < .0001, and no dif-ference between the pronouns in the stressed condition,t (31) = 1.0, p = .32. Additionally, the unstressed pro-nouns were similar in length to those used in Experiment1. The unstressed tokens ofit did not differ from those inthe Experiment 1 objects condition, paired t (31) = .5,

p = .62. The unstressed tokens of that in Experiment 1were on average 159 ms (6.3), only slightly longer thanthe tokens in Experiment 2a, t (31) = 1.91, p = .06.

The mean pitch excursion for pronouns in Experi-ment 2a and 2b are presented in Fig. 6.

An ANOVA for pitch excursion with stress (stressed,unstressed), and pronoun (it, that) as factors revealedonly a main effect of stress, F(1,29) = 48.82, p < .0001,

Fig. 6. Mean pitch excursion for pronouns in Experiments 2a

and 2b.



14/22

due to larger pitch excursions for the stressed pronouns.The duration of words leading up to the pronoun wasalso analyzed. Now was longer in the unstressed condi-tions, for both it, 108.8 ms (4.1) stressed vs. 162.9 ms(2.9) unstressed, and that, 106.6 ms (2.4) stressed vs.133.4 ms (3.4) unstressed. The stressed versions had

longer pauses after the pronoun: pause after it = 22 ms(6.6) stressed vs. 0 ms unstressed, and that = 74 ms(2.8) stressed vs. 52 ms (2.9) unstressed. A ToBI analysisof a randomly selected subset of 40 pronouns fromExperiment 2a (10 of each of the four types) confirmedthese basic differences between the unstressed andstressed pronouns. Each of the 10 unstressed tokens ofit was unaccented, and each token of stressed it(n = 10) and stressed that (n = 10) received an L + H*

accent (Pierrehumbert & Hirschberg, 1990). Six of theunstressed tokens ofthat were unaccented, while four to-kens received an H* accent, due to a slight pause after

the pronoun. The findings for unaccented that are con-sistent with the duration analyses, and confirm thatunaccented that receives more prosodic weight than it.Additionally, the clear difference in accent type betweenthe unstressed and stressed conditions (unaccented/H*

vs. L + H*) indicates that our manipulation of stresswas successful.

For each item in Experiment 2b, one of the tworecordings for it, and one of the two recordings for thatwas used as a carrier phrase (carrier phrases wereequally distributed across the stressed and unstressedconditions). The pronoun was extracted from the secondrecording and spliced into this carrier phrase. A thirdsentence for each of the two pronoun conditions was re-corded using the same stress as the carrier phrase. Thepronoun in these third sentences was spliced into thecarrier phrase, to ensure that all pronouns shared thefeature of being spliced. Co-articulation with the pro-noun made splicing between the pronoun conditionsimpossible. All splicing was done using SoundEdit soft-ware and minimized the presence of odd noises (such asclicks) which can sometimes occur during cross-splicing.While the cross-splicing was done carefully, the sen-tences were not as natural sounding as those in Experi-ment 2a because the prosody of the carrier phrase was

(by design) not always consistent with the prosody onthe pronoun. Despite this inconsistency, we felt it wasimportant to investigate whether the effects we observedin Experiment 2a would persist when only the stress onthe pronoun changed (on average) between conditions.

An analysis of the duration of Put and the space be-fore the pronoun indicated our efforts at splicing weresuccessful. Except for the expected differences betweenthe pronouns, there were no differences between thestressed and unstressed conditions in the length of Put,stressed vs. unstressed for it = 142.0 ms (5.8), 147.0 ms(3.4), and stressed vs. unstressed for that = 96.0 ms

(3.1), 95.4 ms (3.3). An ANOVA for the length of Put

with pronoun type (it/that), stress (stressed/unstressed)as factors, found no effect for stress F(1,31) = .80,

p = .38, and no interaction with pronoun type,F(1,31) = 1.2, p = .29. There were no differences be-tween the stressed and unstressed versions of either pro-noun in the length of the pause after the pronoun,

stressed vs. unstressed for it = 0 ms (0), 0 ms (0), andstressed vs. unstressed for that = 29.0 ms (5.1), 30.5 ms(5.4). Finally, an ANOVA for pitch excursion with stress(stressed and unstressed), and pronoun (it, that) as fac-tors revealed only a main effect of stress,F(1,31) = 25.20, p < .0001, due to a larger pitch excur-sion for the stressed pronouns. A ToBI analysis of a sub-set of 40 pronouns indicated that the unstressed tokensof it were all unaccented (n = 10), and the stressed to-kens of both it (n = 10) and that (n = 10) all receivedthe L + H* accent. The analysis of the unstressed tokensof that indicated that 2/10 were unaccented, and the

remaining eight received the H*

accent. The larger pro-portion of H* accents for unstressed that in Experiment2b is likely due to differences in phrasal prosody inExperiment 2b. In order to be able to splice the pro-nouns, the Now was removed from the critical instruc-tions, and the speaker had to pause slightly after theunaccented pronoun.

Experiments 2a and 2b: Results

We analyzed the referent selection data in the samemanner as Experiment 1, in order to facilitate compari-sons between the experiments. We focus on comparingthe interpretation of the stressed and unstressed pro-nouns, to assess whether they are similarly affected bythe presence of prosodic stress. The results from Exper-iments 2a and 2b were similar. The primary differencebetween the two experiments was a lower baseline oftheme responses in Experiment 2b.

Referent choice

When participants heard instructions such as Put theballoon next to the road. Now put it/that. . ., they typi-cally moved either the theme (the balloon) or the com-posite (both the balloon and road) to the specified

location. On a small number of trials, participants se-lected the goal (e.g., the road), however this representedonly 4% of the data. Due to the low frequency of occur-rence, we will not discuss those responses further exceptto note that they are inconsistent with Kameyamas(1999) complementarity hypothesis, which predicts thatstressed it should prefer goal interpretations.

Replicating the patterns seen in Experiment 1, theunstressed pronouns had different interpretation prefer-ences, with it interpreted as the theme, and that as thecomposite, and the interpretation of both pronounswas affected by the location manipulation. In addition,

the stressed form ofit decreased the proportion of theme



15/22

interpretations, whereas the stressed form ofthat had lit-tle effect on interpretation preferences. Figs. 7A and Bshow the proportion of trials in which participants se-lected the theme, goal or composite for Experiment 2a.

Figs. 8A and B show the proportion of trials in whichparticipants selected the theme, goal or composite for

Experiment 2b.To add to the power of our analysis and compare the

results in Experiments 2a and 2b, we analyzed the datafrom the two experiments together. An ANOVA forthe proportion of theme responses with pronoun (it/that), object location (on top/next to), and stress(stressed/unstressed), as well as the between-subjectsexperiment factor (Experiment 2a/2b) revealed a maineffect of pronoun type, due to significantly more themeresponses for it, F1 (1,30) = 42.74, p < .0001,F2 (1,62) = 248.06, p < .0001. The main effect of loca-tion was significant, F1 (1,30) = 17.24, p < .001,

F2 (1,62) = 14.39, p < .001, due to more theme responsesin the next to condition. The main effect of stress wasdue to significantly more theme responses for the un-stressed pronouns, F1 (1,30) = 7.29, p < .05,F2 (1,62) = 7.0, p < .05. The main effect of Experimentwas marginal in the subjects analysis, and significant inthe items analysis, F1 (1,30) = 4.14, p = .05,F2 (1,62) = 141.5, p < .0001, and was due to fewer themeresponses in Experiment 2b. The pronoun by stressinteraction was significant, F1 (1,30) = 9.68, p < .01,F2 (1,62) = 9.39, p < .01, and was due to a significant de-crease in theme responses for stressed compared to un-stressed versions of it, F1 (1,30) = 9.88, p < .01,

F2 (1,62) = 12.50, p < .001, and no effect of stress forthat, F1 (1,30) = 1.0, p = .34, F2 (1,62) = .55, p = .46.We observed an interaction between pronoun and exper-iment that was significant only in the items analysis,F1 (1,30) = 1.73, p = .2, F2 (1,62) = 15.02, p < .001.This interaction is likely due to the fact that the theme

bias for it was slightly larger in Experiment 2a,F1 (1,15) = 52.31, p < .0001, F2 (1,31) = 121.98,

p < .0001, than Experiment 2b, F1 (1,15) = 9.66,p < .01, F2 (1,31) = 104.63, p < .0001. We also observeda significant interaction between location and experi-ment, F1 (1,30) = 5.56, p < .05, F2 (1,62) = 5.28,

p < .05. This interaction was a result of the fact thatthe location effect was not significant for Experiment2b, F1 (1,15) = 1.92, p = .19, F2 (1,31) = .9, p = .35,whereas for Experiment 2a we observed significantlymore theme responses in the next to than the on top con-dition, F1 (1,15) = 18.26, p < .001, F2 (1,31) = 12.82,

p < .01. A closer look at the response pattern for Exper-iment 2b suggests a hint of a location effect for the goalresponses, with more goal selections in the next to con-dition than the on top condition, however the small num-ber of trials and numerous missing cells makes astatistical comparison impossible.

In summary, we replicated the patterns seen inExperiment 1 for the unstressed pronouns; basic differ-ences in the interpretation of it and that were modulatedby factors that increased the availability of the compos-ite. The data from the stressed pronouns adds to ourunderstanding of these interpretation preferences byshowing that adding stress to it decreases theme inter-

Fig. 7. Experiment 2a: proportion of theme, goal, and composite selections for stressed (str) and unstressed (unst) it and that in thenext to (A) and on top (B) conditions. Grey portion, theme responses; white, composite; black, goal.

Fig. 8. Experiment 2b: proportion of theme, goal, and composite selections for stressed (str) and unstressed (unst) it and that, in the

next to (A) and on top (B) conditions. Grey portion, theme responses; white, composite; black, goal.



16/22

pretations, but stress has no effect on the demonstrativethat. Finally, although the object pairs we used in thisexperiment did not form natural groups as they did inExperiment 1, we observed an almost identical patternof interpretation for the unstressed pronouns as we didin the Experiment 1 objects conditions, and more com-

posite interpretations than in the blocks condition.One reason why participants may have easily interpretedthe mismatching objects as a composite is that they mayhave invented idiosyncratic groupings for the objectpairs. The Experiments 2a and 2b object pairs have morefeatures than those in the blocks condition, thus com-posites may be more likely with the objects. Example 9shows one of the object pairs in Experiment 2a, whichincluded a frog and a dolls leg.

(9) a. Put the leg on top of the frog.b. Now put that next to the boat.

This particular trial elicited 100% composite re-sponses when the pronoun was that (regardless of theobject location manipulation), perhaps because partici-pants associated frogs and legs (as in frog-leg soup) orpossibly because (in the on top condition), the leg bal-anced nicely on top of the frog. Unlike the blocks ofExperiment 1, objects with multiple salient featuresmay be more able to form a conceptual composite,prompting composite interpretations of that.

Experiments 2a and 2b: Conclusions

In summary, we replicated the off-line findings ofExperiment 1 for an overall theme bias for it, and apragmatic effect for both pronouns, due to increasingthe availability of the composite. Additionally, we foundthat stressing it decreased the proportion of theme inter-pretations, due to an increase in the proportion of bothcomposite and goal interpretations. This result supportsthe first hypothesis, that stressing the personal pronounshifts the interpretation away from the focus, and is con-sistent with previous claims that stressed pronouns referto non-focused or less salient entities (Akmajian & Jack-endoff, 1970; Cahn, 1995; Lakoff, 1971; Nakatani, 1997).

However, the dominant interpretation of stressed it wasstill the theme, a finding that is somewhat inconsistentwith the second hypothesis, that stressing a personalpronoun results in a preference for the least salient entity(e.g., the goal). While the basic theme preference castsdoubt on Kameyamas (1999) complementary preferencehypothesis, the small increase in goal responses forstressed it is consistent with her basic claim. The resultsfor the demonstrative showed a different pattern, withno difference between the stressed and unstressed ver-sions of that, and more composite responses comparedto stressed it. These results are inconsistent with the

third hypothesis that the primary difference between un-

stressed it and that is stress. The findings for that insteadsuggest that the demonstrative may be insensitive toadded stress, perhaps because it already receives moreprosodic weight than the personal pronoun, as evidencedbytheH* accents on some of the unstressed tokensofthat.Systematically manipulating the type of accent (unac-

cented, H*, L + H*) could provide insights into the role,if any, of stress on the interpretation of that.

The fact that we observed effects of stress in Experi-ment 2b that were comparable to those in Experiment 2asuggests that the effect of prosody for it in Experiment2a was not primarily due to differences in the intonationof the carrier phrase, but specifically due to the changein the stress on the pronoun. The consistent lack of aneffect of stress for that is likely due to the fact that thatnormally receives more stress than it, and that stress onthe demonstrative pronoun is not interpreted contras-tively as it is for the personal pronoun. Taken together,

the results for unstressed and stressed versions of it,combined with the results for both versions ofthat, showa basic difference between the pronouns, as well as astress effect for it, with unstressed it having the largestnumber of theme interpretations, that (stressed and un-stressed) having the fewest theme interpretations, andstressed it somewhere in-between. The fact that re-sponses to stressed it are in-between those for unstressedit and both versions ofthat, even when the composite ismost competitive with the theme (on top conditions),supports three conclusions: (a) the personal pronounprefers the most salient entity; (b) stressing it weakensthis preference; and (c) the primary difference betweenthe pronouns is not due to stress. Rather, the demon-strative that preferentially refers to composite or task-relevant entities.

Finally, in comparing the results of Experiments 2aand 2b, the numerically lower rate of theme responsesin Experiment 2b is likely due to the fact that the sen-tence prosody was odd, a result of the cross-splicingmanipulation. Additionally, removing Now from thecritical pronoun sentences in Experiment 2b may havedecreased the continuity of the instructions. WhenNow is used as a discourse marker (as it is in these con-texts), it can mark a change of focus in the discourse

(Grosz & Sidner, 1986), or a further development ofthe previous topic (Reichman, 1985). Removing Nowmay have decreased the likelihood that the secondinstruction was interpreted as a continuation of the first,thus mitigating the relative salience of the theme estab-lished in the first instruction, and resulting in a decreasein overall theme selections.

Experiment 3: Discourse coherence

In each of the preceding experiments, the theme (e.g.,

the cup in example 4a) has the highest salience according



17/22

to grammatical role ranking when this first sentence con-cludes. However, the influential centering framework(Grosz et al., 1995) asserts that a within-sentence rank-ing of salience is not as reliable a predictor of pronouninterpretation preferences as a ranking that takes intoaccount cross-sentence coherence. Centering theory pre-

dicts that, in order to maximize discourse coherence,pronouns preferentially select for referents that are mini-mally oblique in the previous sentence and that maintaintopicality across sequences of sentences. A typicalobliqueness ranking is Subject < Direct Object < IndirectObject < Adjuncts. Centering defines the backward-look-ing center as the least oblique element of the previous sen-tence which is repeated in the current sentence. Crucially,according to this definition, the first sentence of a dis-course has no backward-looking center, and thereforethe preferred pronoun interpretation in the second sen-tence of a discourse is only weakly established. Thus, it

is not until the third sentence of a discourse that bothof the centering rules come into play for pronoun inter-pretation preferences. Recall the materials used inExperiment 1, which are repeated in (10). Following10a, the highest ranked entity, the cheese, should bethe preferred referent for a subject pronoun in the fol-lowing sentence, 10b. However, the fact that there is nopreceding linguistic context before 10a, causes predic-tions regarding interpretation of a pronoun in 10b tobe less clear. The possibility that the theme was notadequately focused at the onset of the pronoun may ac-count for the large (up to 40%) number of trials inwhich our participants interpreted the personal pro-noun it as referring to a relatively less salient entitysuch as the composite.

In Experiment 3, we used a two-sentence sequence toestablish the theme as the most salient entity accordingto both grammatical role ranking and Cb-continuity.If it was frequently interpreted as the composite inExperiments 1 and 2 because the theme was not thebackward-looking center, we should observe moretheme interpretations for it when we have two contextsentences, rather than the single context sentence as inExperiments 1 and 2b.

Method

To directly compare the results of Experiment 3with those of Experiment 1, we returned to using objectpairs that were conceptually related (see object list inAppendix A). In addition to manipulating the pronoun(it/that), and the preposition (next to/on top), wemanipulated whether there was a single context sen-tence before the critical pronoun sentence (as in Exper-iments 1 and 2b), or two context sentences (seeexamples 10 and 11). The one-sentence condition (10)differs from the two-sentence condition (11), in that

(11) more clearly establishes the b-sentences theme as

the focus when the sentence containing the critical pro-noun begins:

(10) a. Put the cheese next to the cracker.b. Now put it/that in front of the candle.c. Put the candleholder behind the candle.

d. Now put the candleholder next to the candle.(11) a. Put the candle and the cheese to the right of the

candleholder.b. Now put the cheese next to the cracker.c. Now put it/that in front of the candle.d. Now put the cheese and the candle in front of

the candleholder.

In sentence 10b, when the participant encounters thepronoun, two of the four objects have been mentioned inthe discourse. The cheese is of higher salience than thecracker, because it has been mentioned first, however

it is not the backward-looking center of sentence 10a,because there was no prior sentence. In contrast, in11c, when the participant hears the pronoun, the cheeseis the most salient entity in the discourse because it wasmentioned before the cracker in the previous sentence.The cheese is also the backward-looking center of 11b,because it is the highest ranked (and only) discourse en-tity in 11b that was also mentioned in 11a. The fact thatthe cheese is more clearly established as the focus in (11)predicts a stronger theme preference for it in 11c, than in10b, on some accounts.

The filler trials did not contain pronouns and wereconstructed in the same way as Experiments 1 and 2b.Equating the probability of moving each of the objectson filler trials makes the sequences of instructions lesspredictable. However, an effect of discourse coherenceshould be observable when evaluating the effect of thenumber of context sentences on the interpretation ofthe pronoun.

Results

The results of Experiment 3 are comparable to theoff-line results in the Experiment 1 objects condition.We observed no effect of the focusing manipulation.

Thus, the fact that the theme was not the backward-looking center in Experiments 1 and 2b is not responsi-ble for the high proportion of composite responses. In-stead, it appears to be sensitive to the availability ofthe composite. Figs. 9A and B show the proportion oftheme, goal and composite responses in each of the eightconditions.

As in Experiments 1 and 2b, the location of the themeand goal objects at the beginning of the critical instruc-tion clearly affected the degree to which participantsinterpreted the pronoun as the composite, with far morecomposite interpretations in the on top conditions for

both pronouns.



18/22

As before, we analyze the proportion of theme re-sponses. Less than 4% of the responses were goal selec-tions, thus higher proportions indicate fewer compositeresponses. An ANOVA for the proportion of theme re-

sponses with pronoun (it/that), object location (on top/next to), and focus (2 sentence/3 sentence) as factors, re-vealed a main effect of pronoun, which was due to moretheme responses in the it condition, F1 (1,15) = 47.81,

p < .0001, F2 (1,31) = 324.0, p < .0001. The main effectof object location was due to more theme responses inthe next to condition, F1 (1,15) = 23.97, p < .001. Therewas no effect of focus, F1 (1,15) = .002, p = .97,F2 (1,31) = .00, p = .99. The pronoun factor interactedwith object location, F1 (1,15) = 4.93, p < .05,F2 (1,31) = 23.25, p < .0001. Separate ANOVAs indi-cated that this interaction was due to a larger locationeffect for that than for it. The effect for it was small,98% (SD = 2%) theme responses for next to vs. 88%(4.8%) for on top, F1 (1,15) = 7.8, p < .05,F2 (1,31) = 9.2, p < .01, whereas the effect for that wasmuch larger, 52% (5%) theme responses for next to vs.23% (5%) theme responses for on top,F1 (1,15) = 15.76, p < .01, F2 (1,31) = 38.76, p < .0001.The location by focus interaction was not significantfor it in either the subjects or items analysis,F1 (1,15) = .72, p = .41, F2 (1,31) = .81, p = .37, norwas it significant for that, F1 (1,15) = 2.89, p = .11,F2 (1,31) = 1.09, p = .30.

Experiment 3: Conclusions

In summary, the results of Experiment 3 clearly rep-licate the results seen in Experiments 1 and 2b for the 2-sentence condition. We replicated the theme bias for it aswell as the main effect of location, suggesting that bothpronouns are affected by increasing the availability ofthe composite. Additionally, we observed a significantpronoun by location interaction in this condition, dueto a slightly weaker location effect for it than that, indi-cating that that was particularly sensitive to the avail-ability of a composite in the scene. Crucially, the

number of context sentences did not affect interpreta-

tion. Thus the frequent interpretation of it as the com-posite is not due to failure to establish the theme asthe backward-looking center in Experiments 1 and 2b.

General conclusions

In this series of experiments we investigated the onand off-line interpretation of the personal pronoun itand the demonstrative pronoun that. In Experiment 1,we found a difference between the pronouns in boththe eye movement and the referent selection data, withit preferentially referring to the theme, and that referringto the composite. Preferences for both pronouns weremodulated by the availability of the composite, asmanipulated by the location of the two most recentlymentioned objects (e.g., on or next to). On theassumption that it refers to the most salient entity in adiscourse, the increase in composite interpretations forit in some conditions indicates that a referent withouta linguistic antecedent can compete with the most salientlinguistically introduced referent. Consistent with thisconclusion is the fact that composite interpretationswere overwhelmingly preferred to goal interpretationsin every condition. These findings are consistent withsalience-based approaches, so long as linguisticallyintroduced antecedents are not given priority overnon-li

Beyond Salience - Interpretation of Personal

Documents

Transcript of Beyond Salience - Interpretation of Personal