Enhancing Natural Language Understandingthrough Cross-Modal Interaction:

Meaning Recovery from Acoustically Noisy Speech

Ozge AlacamDepartment of Informatics

University of Hamburg, Hamburg, Germanyalacam@informatik.uni-hamburg.de

Abstract

Cross-modality between vision and lan-guage is a key component for effectiveand efficient communication, and humanlanguage processing mechanism success-fully integrates information from variousmodalities to extract the intended mean-ing. However, incomplete linguistic in-put, i.e. due to a noisy environment, isone of the challenges for a successful com-munication. In that case, incompletenessin one channel can be compensated byinformation from another one (if avail-able). In this paper, by employing a visual-world paradigm experiment, we investi-gated the dynamics between syntacticallypossible gap fillers for incomplete Germansentences and the visual arrangements andtheir effect on overall sentence interpreta-tion.

1 Introduction

In recent years, a growing body of literature hasinvestigated how and to what extent cross-modalinteraction contributes to natural language under-standing. Human language processing system in-tegrates information from various modalities toextract the meaning of the linguistic input ac-curately, but the contribution of cross-modalityto a successful communication goes beyond it.It facilitates early reference resolution while thesentence unfolds and allows disambiguation evenwithout realizing that another (linguistic) interpre-tation would be possible, e.g. see (Altmann andMirkovic, 2009; Knoeferle et al., 2005; Tanen-haus et al., 1995). Furthermore, it also preparesthe grounds for the re-construction of the mean-ing from noisy/missing input. When the environ-ment is noisy, or the communication partner suf-fers from a motor or cognitive impairment, text

completion/prediction becomes a crucial elementof a communication. Instead of waiting for orrequesting spoken input, combining the uncertaininformation from the linguistic channel with in-formation from the visual one increases the flu-ency and the effectiveness of the communication(Garay-Vitoria and Abascal, 2004).

In this study, by conducting an experiment withhuman-subjects, we address the problem of com-pensating the incompleteness of the verbal chan-nel by additional information from visual modal-ity. Investigating how humans reconstruct themeaning from a noisy data provides insights abouthow to incorporate human-like processing intocommunication systems. The psycholinguistic ex-periments help us to understand baseline prefer-ences and the underlying mechanism of gap con-struction processes for meaning extraction. Thiscapability for multi-modal integration can be avery specific yet crucial feature in resolving ref-erences and/or performing commands for i.e. ahelper robot that aids people in their daily activ-ities.

2 Meaning Recovery

The task of extracting meaning from a noisy in-put has been widely addressed by uni-modal ap-proaches. In a uni-modal way, re-construction canbe guided by e.g. morphological, syntactic, andsemantic properties. In that case, a probability ofa syntactic category in a certain context can be ob-tained from a language model (Asnani et al., 2015;Bickel et al., 2005). For example, using N-gramsis a popular method for this task since they pro-vide very robust predictions for local dependen-cies. However, their power is less when it comesto dealing with long-range dependencies. On theother hand, as several studies (Mirowski and Vla-chos, 2015; Gubbins and Vlachos, 2013) show, alanguage model employing the syntactic depen-dencies of a sentence brings the relevant contexts

closer. Using the Microsoft Research SentenceCompletion Challenge (Zweig and Burges, 2012),Gubbins and Vlachos (2013) have showed that in-corporating syntactic information leads to gram-matically better options for a semantic text com-pletion task. Semantic classification (e.g. ontolo-gies) and clustering can also be used to derive pre-dictions on the semantic level for meaning recov-ery. However, when it comes to the descriptionof daily activities, contextual information comingfrom another modality would be more beneficial,since linguistic distributions alone could hardlyprovide enough clues to distinguish the action ofbringing a pan from bringing a mug, which is acrucial difference for e.g. helper robots.

Cross-modal integration of two modalities canbe addressed by various methods in a range fromsimply putting all features from both modali-ties together and then train a model to learnthe associations, to more complex structures,e.g. relating uni-modal features from sev-eral modalities on a conceptual level by usingcommon representations. Considering that thetask of meaning extraction may benefit fromnot only low-level but also high-level knowl-edge representations, one meaningful methodwould be to utilize a triplet notation, consistingof (argument, relation type, predicate) whererelation type is one of a predefined set of ac-cepted relations, such as AGENT or THEME whilePredicate and Argument are tokens of the in-put sentence. Within this framework, the re-construction of content words can be formalizedas recovering/predicting the predicates or argu-ments of a sentence. To put it simply, a sentencelike “the woman carries ....” can be formulatedinto two triplets; (womani,AGENT, carry) and(unknowni, THEME, carry). Here the task is todetermine the unknown entity which has directlyrelated to the carry action and indirectly to theagent woman. In case the contextual informationprovided by the visual environment contains ad-ditional information (e.g. a scene that depicts awoman with a grocery bag), the missing part canbe successfully filled.

Salama et al. (2018) address the problem of in-complete linguistic input referring to daily envi-ronment context by utilizing a context-integratingdependency parser. Their focus was to recovercontent words like nouns, adjectives and verbsgiven the contextual features (e.g. object prop-

erties, spatial relations among the objects or the-matic roles). The results indicate that giving astrong influence to contextual information helps tofill a majority of gaps correctly.

While re-construction of content words ismostly about finding out either the argument or thepredicate based on the relation between each other,re-construction of grammatical words is to deter-mine the relation between argument and predi-cate. Furthermore, re-construction of grammaticalwords could be more challenging since they tendto occur with higher frequencies than the contentwords, yielding a very small type/token ratio (i.e.weaker a collocational relationship) that makes thereconstruction of them based on only linguisticinformation more difficult. Although this is be-yond the scope of the current paper, it should benoted that a full-fledged cross-modal meaning re-covery system is dependent on a success of visualrelation extraction component as well. The state-of-art computer vision systems can be consideredmore effective to extract spatial relations amongobject and object properties compared to relationsbetween the actors and their actions.

3 Situated Language Comprehension ina Noisy Setting

The noise in communication could be originatedfrom various channels and sources. First of all,it can be a linguistic noise (e.g. spelling mis-takes, complex attachments), or visual ambiguities(e.g. clutter of the environment, occlusions) or anacoustic noise.

The issues of how to comprehend noisy lin-guistic input and reconstruct the intended meaninghave been addressed by both psycholinguistic andcomputational line of research (e.g. (Levy, 2011,2008)).

According to a noisy-channel account, thatmainly focus on linguistic noise, the sentencecomprehension mechanism integrates all the in-formation (at the syntactic, semantic and dis-course level) from the existing words and use thislinguistic evidence to predict the missing partsand infer the possible meaning (Gibson et al.,2013). Several studies have shown that in caseof higher degrees of syntactic complexity, hu-mans tend to choose an interpretation which isin line with the concurrent visual information orgeneral world knowledge, even though this inter-pretation requires to accept grammatically unac-

ceptable syntactic structures (Johnson and Char-niak, 2004; Christianson et al., 2001; MacWhin-ney et al., 1984). Cunnings (2017)’s study on lan-guage learners also indicated when the perceiverprocesses (syntactically) noisy linguistic input, theother linguistic and non-linguistic constraints areprioritized compared to syntactic ones.

Based on noisy-channel framework, Levy(2008) proposes a probabilistic model of languageunderstanding regarding situations where there areuncertainty about word-level representations. Headdresses the problem in two different levels; aglobal inference that can be reached after process-ing the entire input, and incremental inference thatis formed (usually) word-by-word the sentenceunfolds. The main contribution of the proposedmethod is that it takes into account the prior andposterior probabilities calculated based on bothlinguistic and non-linguistic evidence, includinge.g. the expectations about speaker’s grammaticalcompetence or about the environmental conditionthat can hinder the speech signals.

Gibson et al. (2013) describes language under-standing as rational integration of noisy evidenceand semantic expectations. In their study, theytest their predictions by conducting reading ex-periments, in which mostly the prepositions inthe sentences were altered (by deletion or inser-tion) keeping content-word same across condi-tions. For example, an ungrammatical sentence“The mother gave the candle the daughter” canbe easily treated as plausible by inserting to be-fore “the daughter”. The higher prior probabilityof the latter version of the sentence compared tothat of the former one pulls the sentence meaningtowards itself.

4 Negation Processing

One interesting question regarding the task ofmeaning recovery is how to recover a meaningcommunicated with a sentence that involves un-clear negated statement.

Since negation is considered as a higher orderabstract concept, it has its own uniqueness as agrammatical category. Identifying the scope andfocus of negation is one of the challenging issuesthat gets particular attention from the NLP com-munity (e.g. SEM 2012 shared task, Morante andBlanco (2012)). From a psycholinguistic perspec-tive, the core discussion lies around whether bothnegated and actual situation of content is simu-

lated or only the actual one. However, regardlessof how this process happens, the literature agreeson that sentences containing negation are harder tointerpret than affirmative sentences (Orenes et al.,2014; Khemlani et al., 2012; Kaup et al., 2006;Ludtke and Kaup, 2006; Carpenter and Just, 1975;Clark and Chase, 1972).

It has been conclusively shown that a negativesentence is processed by first simulating the pos-itive argument. For example, after reading a neg-ative sentence “The bird was not in the air”, aresponse to image that depicts a flying bird wasfaster than to a image of a bird at rest, (Zwaan,2012). In addition to an overall processing diffi-culties that negation entails, it has been also shownthat it is only integrated into the sentence meaningat a later point (Ludtke et al., 2008).

On the other hand, there are also some evi-dence that indicates that when negation is sup-ported by right contextual support, the positive ar-guments is no longer need to be represented, yield-ing faster verification compared to no-context sit-uations (Tian et al., 2016; Dale and Duran, 2011;Nieuwland and Kuperberg, 2008).

5 Experiment

This study focuses on humans’ preferences for thereconstruction of unclear sentence parts (in Ger-man) by using visual-world paradigm. Moreover,the effect of contextual information on situatedreference resolution and on gap re-constructionhas been also manipulated by restricting the affor-dances of the locative object in one of the visualarrangements.

Gibson et al. (2013) list four different criteriaof language processing system that have an im-pact on meaning recovery; (i) how close the literalsentence is to the plausible alternative, (ii) whatkind of change is involved (insertion or deletion),(iii) the expectations about the corruption (noise-rate), and (iv) the plausibility of implausible sen-tences based on context or speaker-based. Basi-cally by keeping all the criteria described by Gib-son et al. (2013) constant, we focus, in this ex-periment, on obtaining prior probabilities of threetypes of grammatical words; two common prepo-sition of location (on and next to) and negation par-ticle (not). All sentences are syntactically plau-sible regardless from which focus-of-interest gapfiller is used and their semantic plausibility isdependent on the information coming from the

visual-world. Here in this current paper, we fo-cus more on how meaning recovery is affected bynegation, instead of detailed discussion into nega-tion processing. Thus we kept the focus of nega-tion constant among conditions, and the scene hasbeen designed to have low referential competition(i.e. there are two tables in the scene, makingthe decision a binary task instead of a multinomialone).

The task is simply, hearing a sentence that com-municates the intended meaning and process it andextract the meaning as close as possible to the in-tended one (Shannon, 1948). The goals that needto determined are;

• the re-construction of the gap-word

• full-sentence interpretation (“which objectneeds to be moved, and where to”)

5.1 Participants20 students (native speakers of German) partici-pated in the experiment (Mean age = 23.8, SD =3.1). They were paid or given a course credit toparticipate. The entire experiment took approxi-mately 45 minutes for each participant includingthe familiarization and instruction sessions.

5.2 MaterialLinguistic Material. In their complete form with-out a gap, all sentences have the same structureexcept the negation/preposition part (NEG/PP) asgiven below. The sentences start with a verb inan imperative form preceding an object (NP) anda prepositional phrase that specifies the goal loca-tion (PP). Then the sentence continues with a dis-fluency (umm) and a repair/complement part con-sisting of a negation or one of the two prepositionof location. Our focus-of-interest gap fillers are(nicht (not), auf(on), neben (next to). These arechosen since they can fill the same position inter-changeably.

• Stell den Becher auf den Tisch, umm[auf/nicht/neben] den blauen.put the mug on the table, umm [on/not/nextto] the blue one.

The choice of filler-word given the visual in-formation determines which object that the re-pair/complement part is attached to. In this set-ting, the repair/complement may have three differ-ent syntactic roles; referring back to the OBJECT

which is the mug (with not), referring back to theADVERBIAL which is the table (with both onand not) or providing new complementary AD-VERBIAL which is an another mug (with nextto). Due to filling different roles, all possible lin-guistic interpretations require different parsing re-sults. In all cases, the object referred to in therepair/complement part shares either the property(e.g. blue) or the object class (e.g. mug) with thetarget object or location.

Pre-prossessing of the spoken material. Thesentences were recorded by a male native speakerof German at a normal speech rate. Intona-tional differences between different linguistic en-tities have been found to have a significant effecton reference resolution (Coco and Keller, 2015;Snedeker and Trueswell, 2003). Therefore, weavoided unequal intonational breaks that may biasthe interpretation. The breaks separating phraseswere equalized.

A constant background noise (a sound record-ing from a restaurant) was added to an entire spo-ken sentence with the Audacity software 1. In or-der to mask the target word completely, the vol-ume of the NEG/PP part starting from the inter-jection (umm) was gradually decreased till the endof the gap-word. Concurrently, the volume of thebackground noise was increased during this seg-ment.

Scenes. In order to accommodate the intendedinterpretation(s) and to eliminate others, the ob-ject properties and their spatial relations amongeach other have been systematically manipulatedfor each scene. Although, other many more differ-ent visual arrangements could be possible, for thesake of systematicity, we have narrowed our visualconditions down to five scene arrangements, seeFigure 1. Scene-1 conveys all possible interpreta-tions for all the focus-of-interest fillers. Scene-2Aand Scene-2B allows only on and not. However,the availability of the location signaled by on isoccupied by another object in Scene-2B. The lasttwo visual arrangements allows only one interpre-tation; signaled by not in Scene-3A and by next toin Scene-3B. The number of objects in the sceneswas limited to eight and one additional object hasbeen used in Scene-2B. For each visual condition,six different visual scene were designed resulting30 main-trial scene. The 2D visual scenes were

1http://www.audacityteam.org/ - retrieved on 21.11.2018

created with the SketchUp Make Software 2.To prevent participants’ associating the focus-

of-interest gap fillers with a particular visual ar-rangement, additional slightly changed sentencesand scenes were introduced as filler items.

5.3 Procedure

Using the visual-world paradigm, we presentedparticipants visual scenes with accompanying spo-ken sentences. We employed a simple “look-and-listen” experiment following clicking tasks to getuser’s preferences. The experiment started withfilling out the written consent and demographicdata form. Afterwards, the instructions were givenin written format, preceding the 3 familiarizationtrials.

The participants were instructed that multi-modal stimuli always contain some backgroundnoise, and at one point, one word will be impossi-ble to hear. Then they are expected to choose thegap-filler word and click on the target object andlocation communicated in the sentence. It was toldthat target object is always located on the middlestand and needs to be moved to one of the whitetrays on the scene located on various places. Inorder to be able to separate the task of identifyingobject from identifying location, target objects andlocations are presented in a specific layout.

The stimuli were displayed on an SR Eyelink1000 Plus eye tracker integrated into a 17 mon-itor with a resolution of 1280 x 1024 pixels. Weutilized a total of 53 visual displays with accompa-nying spoken utterances (3 familiarization, 30 testtrials and 20 fillers). Each trial began with a a driftcorrection and the presentation of a simple fixa-tion cross for 2 sec, located at the middle-bottomof the screen. Afterwards, a 5 sec of visual pre-view before the onset of the spoken sentence wasgiven. The preview gives a comprehender time toencode the visual information in advance of thelinguistic information being presented. So, visualattention is intended to be free of recognizing theobjects of the visual context during language pro-cessing. Then, the spoken sentence was presentedaccompanying the visual stimulus. A trial ended2 sec after the offset of the sentence. Participantswere asked to examine the scene carefully and at-tend the information given in the audio. The orderof stimuli was randomized for each participant.

After the sentence is completed, the scene dis-

2http://www.sketchup.com/ - retrieved on 06.05.2018

appears and the participants are asked to click theirpreference for the gap position among five op-tions. They were also informed about that the gapcould be accurately filled by more than one op-tion. These options are “nicht (not)”, “neben (nextto)”, “auf (on)”, “mit (with)”, and “den/das/die(the)3”. Whereas the focus-of interest gap-fillersare syntactically acceptable for the gap posi-tion, two other grammatical words were providedamong the options as distractors; mit (with) andden/das/die (the). In German, the preposition mit(with) requires a dative object, therefore the gen-der of the following article should be differentthan the nominative or accusative forms of the ar-ticle in the repair/complement part. Furthermore,den/das/die (the) can be understood in two ways,either as a repetition of the definite article or as arelative pronoun. In the former case, as a gap filler,it does not provide any additional information. Onthe other hand, the lack of relative clause verbmakes the second interpretation unacceptable.

After the preference has been explicitly made,the scene appears again so that the participant canclick on the scene to answer two questions respec-tively; which object is the target? and where is thetarget location?.

Although a time-course analysis of fixateditems and locations when the sentence unfoldsis very relevant in understanding the underlyingmechanisms of language processing, in this paper,we narrow down our scope into participants’explicitly made choices after each multi-modalstimulus.

Our hypothesizes are listed as

- Syntactically all gap positions require oneinsertion to correctly accommodate the in-tended meaning, however unlike prepositionof locations, negation operation is consideredas a high-level (abstract) concept. Therefore,the sentences with on and next to should bemore easier to disambiguate, therefore morepreferred compared to ones with not.

- Conceptual information, i.e. target location’sbeing not available as illustrated in Figure 1emay force to change the interpretation, ac-cordingly the preference, from on to not.

3The respective article/relative pronoun was shownamong options in accordance with the grammatical genderof the noun it modifies

(a) Scene-1: on, next to, not

.(b) Scene-2A: on, not (c) Scene-2B: not, on

(d) Scene-3A: only not (e) Scene-3B:only next to

Figure 1: Sample scenes that illustrate five different visual manipulations

6 Results

In this section, the gap-filler preferences and theglobal sentence interpretation by analyzing accu-rately chosen object given their preference havebeen reported. Figure 2 shows the distribution ofthe preferences for each visual condition. In total,participant preferences for 600 trials (20 partici-pant * 30 scene) were taken into account.

Gap Construction Preferences. The visualcondition Scene-1 was designed to analyze user’sgeneral tendency among three focus-of interestgap fillers, since all are equally plausible w.r.t. thevisual context. In this condition, next to was pre-ferred more in 43.9% of the trials compared to on

(30.8%) and not (20.6%). The other distractor op-tions were preferred only in 3.7% of trials. The re-sults of a Friedmans ANOVA indicated that prefer-ence rates for the three focus-of interest gap fillerssignificantly differs (χ2(2) = 8.95, p < .001).Wilcoxon tests were used to follow up this find-ing. It seems that this difference among groups isoriginated by the difference between not and nextto (z − score = −2.23, p < .0167), with a Bon-ferroni correction).

The analysis on whether the participant couldchoose the object and the location in line withtheir explicitly made preference also demonstratedthat all target objects are correctly identified. Thisresult is highly expected, considering that the

PP/NEG part does not carry relevant informationfor the target identification in this visual setting.On the other hand, regarding the location, while100% of the participants, who chose next to, cor-rectly determine the target location, which is inline with their preference, this accuracy score is90.9% for on and it drops drastically to 71.4% fornot.

Figure 2: Preference distributions regarding eachvisual condition

The Effect of Contextual Cues. Whether theavailability of a location signaled by one of thepossible gap fillers has an effect on the preferenceshas been investigated by a mixed-design ANOVAcomparing the number of preferred option acrosstwo visual arrangements; Scene-2A and Scene-2B. In these conditions, on and not are the twoonly semantically plausible gap fillers. The pro-portion results indicated that when the two lo-cations are equally available (Scene-2A), partici-pants prefer more on as a gap filler (57.5%), andthe option not was chosen in only almost 13.3%of the trials. On the other hand, while the tar-geted location referred by the sentence with on re-pair is occupied (Scene 2B), then the participants’tendency to prefer not increases considerably by21%. The preference of next to stays almost thesame across the conditions.

The results of the ANOVA indicated no maineffect of the visual condition (p > .05). How-ever, the main effect of Preference was signifi-cant (F (2, 38) = 8.642, p = .001). In gen-eral, on has been preferred more compared to notand neben; (F (1, 19) = 10.92, p = .004) and(F (1, 19) = 11.61, p = .003) respectively. Re-garding our research question, the interaction ef-fect between the visual condition and the prefer-ence is the relevant one, and it displays a signif-icant interaction (F (2, 38) = 7.79, p = .001).

This indicates that the preference tendencies sig-nificantly differed in Scene-2A and Scene-2B. Tobreak down this interaction, contrasts were per-formed comparing each level of focus-of-interestpreferences across two scene types. These re-vealed significant interactions when comparing onand not, (F (2, 38) = 18.98, p < .001). Look-ing at the interaction graph in Figure 3, this sug-gests that when the target location signaled byon is occupied, participants looks for alternativesand ending up with only other available interpre-tation in line with not. Moreover, the contrastbetween not and next to was significant as well,(F (2, 38) = 5.30, p < .05).

Figure 3: Mean number of preferred focus-of-interest fillers across Scene-2A and Scene-2B

Preferences under Restricted Conditions.The last comparison focuses on the cases, in whichthe visual arrangements and the properties of theobjects only allow one interpretation. Scene-3Afavors only the interpretation which is in line withthe use of not as a gap filler. Yet, only in 53.3%of the trials correct option has been chosen as gapfiller. Despite their conflict with the visual world,other gap fillers have been chosen in a consider-able amount; next to (16.7%), on (10.8%), with(5.8%) and the/that (11.7%). On the other hand,Scene-3A syntactically allows only next-to. Theresults showed that in 72.5% of the trials, par-ticipants preferred next to. The comparison be-tween the number of correct preferences acrosstwo visual condition revealed that on average, par-ticipants make more correct choices when theysee Scene-3B (M = 4.35, SE = 1.72) comparedto Scene-3A (M = 3.20, SE = .41), (t(19) =−2.31, p < .05).

7 Conclusion

In this study, by systematic manipulation of visualscene, we have obtained prior expectations regard-ing two locative prepositions and negation parti-cle and we have also demonstrated how contextualcues pull an interpretation towards one side.

In order to accommodate different interpreta-tions, five different visual arrangements have beenutilized. Although our investigations into this areaare still ongoing, the results could be a usefulaid for developing models of situated natural lan-guage understanding that aims to account noisydata comprehension and meaning recovery. In thisstudy, we particularly tried to put some spotlightinto the special case “negation” as well.

The results indicate that when the visual worldsupports all interpretations, people have tendencyto choose next to as a gap filler, that entails therepair/complement part referring to another ob-ject, which is not mentioned in the sentence be-fore. Their second preference is to attach therepair part to the prepositional phrase (ADVER-BIAL) by choosing on as gap filler. This selec-tion also inherently assumes that the repair partis an affirmative statement. On the other hand,even in the cases where not is the only semanti-cally plausible option as gap filler, the participantsshowed hesitance to choose it. This results are alsoin line with noisy-channel framework. A sentencewith a gap is more harder to process compared toa complete sentence, since it requires at least twosub-tasks to be performed; predicting the gap-fillergiven the context and then confirming the inferredmeaning. While the spatial relations like next toand on are easily graspable from an image, a neg-ative statement requires additional operation to ac-count for actual situation, that’s why the listen-ers may prefer to override contextual expectationsand stick to more easy-to-process one even it se-mantically, and sometimes syntactically creates aconflict (Ferreira, 2003). However, this preference(choosing on over not) still seems to be affected bythe contextual cues like the availability of a targetlocation.

It should nonetheless be acknowledged that thesystematicity that we had to follow to single outall other effects becomes a limitation for general-ization, thus further research is needed to betterunderstand first the dynamics between the prefer-ence and the visual arrangements and second thedynamics between negation in detail and contex-

tual cues. Moreover, none of the visual manipula-tions in this study was designed to address to ex-plain the difference between choosing next to andon. Another set of experiments with reversed or-der; the first PP with a next to and the complementpart with on would help us to gain some insightson this issue.

Acknowledgments

This research was funded by the German Re-search Foundation (DFG) in the project Cross-modal Learning, TRR-169.

ReferencesGerry TM Altmann and Jelena Mirkovic. 2009. Incre-

mentality and prediction in human sentence process-ing. Cognitive science, 33(4):583–609.

Kavita Asnani, Douglas Vaz, Tanay PrabhuDesai,Surabhi Borgikar, Megha Bisht, Sharvari Bhosale,and Nikhil Balaji. 2015. Sentence completion us-ing text prediction systems. In Proceedings of the3rd International Conference on Frontiers of Intelli-gent Computing: Theory and Applications (FICTA)2014, pages 397–404. Springer.

Steffen Bickel, Peter Haider, and Tobias Scheffer.2005. Learning to complete sentences. In EuropeanConference on Machine Learning, pages 497–504.Springer.

Patricia A Carpenter and Marcel A Just. 1975. Sen-tence comprehension: a psycholinguistic process-ing model of verification. Psychological review,82(1):45.

Kiel Christianson, Andrew Hollingworth, John F Hal-liwell, and Fernanda Ferreira. 2001. Thematic rolesassigned along the garden path linger. Cognitivepsychology, 42(4):368–407.

Herbert H Clark and William G Chase. 1972. Onthe process of comparing sentences against pictures.Cognitive psychology, 3(3):472–517.

Moreno I Coco and Frank Keller. 2015. The interac-tion of visual and linguistic saliency during syntac-tic ambiguity resolution. The Quarterly Journal ofExperimental Psychology, 68(1):46–74.

Ian Cunnings. 2017. Interference in native and non-native sentence processing. Bilingualism: Lan-guage and Cognition, 20(4):712–721.

Rick Dale and Nicholas D Duran. 2011. The cognitivedynamics of negated sentence verification. Cogni-tive science, 35(5):983–996.

Fernanda Ferreira. 2003. The misinterpretation ofnoncanonical sentences. Cognitive psychology,47(2):164–203.

Nestor Garay-Vitoria and Julio Abascal. 2004. A com-parison of prediction techniques to enhance the com-munication rate. In ERCIM Workshop on User In-terfaces for All, pages 400–417. Springer.

Edward Gibson, Leon Bergen, and Steven T Pianta-dosi. 2013. Rational integration of noisy evidenceand prior semantic expectations in sentence interpre-tation. Proceedings of the National Academy of Sci-ences, 110(20):8051–8056.

Joseph Gubbins and Andreas Vlachos. 2013. Depen-dency language models for sentence completion. InProceedings of the 2013 Conference on EmpiricalMethods in Natural Language Processing, pages1405–1410.

Mark Johnson and Eugene Charniak. 2004. A tag-based noisy channel model of speech repairs. InProceedings of the 42Nd Annual Meeting on As-sociation for Computational Linguistics, ACL ’04,Stroudsburg, PA, USA. Association for Computa-tional Linguistics.

Barbara Kaup, Jana Ludtke, and Rolf A Zwaan. 2006.Processing negated sentences with contradictorypredicates: Is a door that is not open mentallyclosed? Journal of Pragmatics, 38(7):1033–1050.

Sangeet Khemlani, Isabel Orenes, and Philip NJohnson-Laird. 2012. Negation: A theory of itsmeaning, representation, and use. Journal of Cog-nitive Psychology, 24(5):541–559.

Pia Knoeferle, Matthew W Crocker, Christoph Scheep-ers, and Martin J Pickering. 2005. The influ-ence of the immediate visual context on incre-mental thematic role-assignment: Evidence fromeye-movements in depicted events. Cognition,95(1):95–127.

Roger Levy. 2008. A noisy-channel model of rationalhuman sentence comprehension under uncertain in-put. In Proceedings of the Conference on EmpiricalMethods in Natural Language Processing, EMNLP’08, pages 234–243, Stroudsburg, PA, USA. Associ-ation for Computational Linguistics.

Roger Levy. 2011. Integrating surprisal and uncertain-input models in online sentence comprehension: for-mal techniques and empirical results. In Proceed-ings of the 49th Annual Meeting of the Associationfor Computational Linguistics: Human LanguageTechnologies-Volume 1, pages 1055–1065. Associ-ation for Computational Linguistics.

Jana Ludtke, Claudia K Friedrich, Monica De Filippis,and Barbara Kaup. 2008. Event-related potentialcorrelates of negation in a sentence–picture verifi-cation paradigm. Journal of cognitive neuroscience,20(8):1355–1370.

Jana Ludtke and Barbara Kaup. 2006. Context effectswhen reading negative and affirmative sentences. InProceedings of the 28th annual conference of thecognitive science society, volume 27, pages 1735–1740. Lawrence Erlbaum Associates Mahwah, NJ.

Brian MacWhinney, Elizabeth Bates, and ReinholdKliegl. 1984. Cue validity and sentence interpreta-tion in English, German, and Italian. Journal of ver-bal learning and verbal behavior, 23(2):127–150.

Piotr Mirowski and Andreas Vlachos. 2015. Depen-dency recurrent neural language models for sentencecompletion. arXiv preprint arXiv:1507.01193.

Roser Morante and Eduardo Blanco. 2012. * sem 2012shared task: Resolving the scope and focus of nega-tion. In * SEM 2012: The First Joint Conferenceon Lexical and Computational Semantics–Volume 1:Proceedings of the main conference and the sharedtask, and Volume 2: Proceedings of the Sixth In-ternational Workshop on Semantic Evaluation (Se-mEval 2012), volume 1, pages 265–274.

Mante S Nieuwland and Gina R Kuperberg. 2008.When the truth is not too hard to handle: An event-related potential study on the pragmatics of nega-tion. Psychological Science, 19(12):1213–1218.

Isabel Orenes, David Beltran, and Carlos Santamarıa.2014. How negation is understood: Evidence fromthe visual world paradigm. Journal of Memory andLanguage, 74:36–45.

Amr Rekaby Salama, Ozge Alacam, and WolfgangMenzel. 2018. Text completion using context-integrated dependency parsing. In Proceedings ofThe Third Workshop on Representation Learning forNLP, pages 41–49.

Claude Elwood Shannon. 1948. A mathematical the-ory of communication. Bell system technical jour-nal, 27(3):379–423.

Jesse Snedeker and John Trueswell. 2003. Usingprosody to avoid ambiguity: Effects of speakerawareness and referential context. Journal of Mem-ory and language, 48(1):103–130.

Michael K Tanenhaus, Michael J Spivey-Knowlton,Kathleen M Eberhard, and Julie C Sedivy. 1995.Integration of visual and linguistic informationin spoken language comprehension. Science,268(5217):1632–1634.

Ye Tian, Heather Ferguson, and Richard Breheny.2016. Processing negation without context–whyand when we represent the positive argument. Lan-guage, Cognition and Neuroscience, 31(5):683–698.

Rolf A Zwaan. 2012. The experiential view oflanguage comprehension: How is negation rep-resented? Higher level language processes inthe brain: Inference and comprehension processes,page 255.

Geoffrey Zweig and Chris JC Burges. 2012. A chal-lenge set for advancing language modeling. In Pro-ceedings of the NAACL-HLT 2012 Workshop: WillWe Ever Really Replace the N-gram Model? On theFuture of Language Modeling for HLT, pages 29–36. Association for Computational Linguistics.