Costs and gains of using Christine - uni-saarland.demirjana/papers/Sekicki,Ankener,Staudte... ·...
Transcript of Costs and gains of using Christine - uni-saarland.demirjana/papers/Sekicki,Ankener,Staudte... ·...
Mirjana Sekicki
Christine Ankener
Maria Staudte
+ the gaze cue
Costs and gains of using
visual context for
referent prediction
Costs and gains of using visual context for referent prediction
Shannon (1948)
• Comprehension difficulty is relative to the predictability of a word in a given context.
• linguistic expressions that are highly predictable, in a given context, convey less info than those which are surprising.
2
Costs and gains of using visual context for referent prediction
Shannon (1948)
• Comprehension difficulty is relative to the predictability of a word in a given context.
• linguistic expressions that are highly predictable, in a given context, convey less info than those which are surprising.
linguistic
script knowledge
visual
3
Costs and gains of using visual context for referent prediction
We were curious …
• cognitive load at the different stages of prediction
1. prediction making
2. prediction (dis)confirming
we were curious…
The woman irons the t-shirt.
• Quantification of the role of visual context in prediction making
4
Costs and gains of using visual context for referent prediction
• We conducted eye-tracking experiments in the visual world and measured:
• Cognitive Load
• prediction forming
• prediction (dis)confirming
• Anticipatory eye-movements
5
Costs and gains of using visual context for referent prediction
The Index of Cognitive Activity
• Wavelet analysis on the pupil dilation record removes all large oscillations and retains only the small and rapid dilations linked to cognitive load (Marshall, 2000)
• Sensitive to Cognitive load induced by language processing (Demberg & Sayeed, 2016)
6
So, we conducted:
1. Ling Study
2. Vis Study
7
Table 1: Sample items and corresponding offline results.
Item noun condition plausibility cloze probability%M (SD) M (SD)
(1) The woman irons soon the a) t-shirt plausible 1.12 (0.68) 13.67 (18.06)b) sock possible 2.76 (2.17) 0.16 (00.54)c) armchair anomalous 6.35 (1.43) <.01
(2) The woman describes soon the a) t-shirt possible 1.65 (1.50) NAb) sock possible 1.90 (1.80) NAc) armchair possible 1.89 (1.78) NA
sure sensitivity. Besides traditional methods, such as readingand reaction time measurements, we also applied a relativelynovel measure: the Index of Cognitive Activity (ICA). ICAis able to capture task related pupil jitter, linked to cognitiveload and predictability (Demberg & Sayeed, 2016). One ofour aims was to examine if ICA can be used to assess dif-ferences in cognitive load induced by plausibility and pre-dictability variation. The advantage of ICA lies in the possi-bility to have the listener view things other than the word sheis currently processing. This would enable the exploration ofvisual context effects on cognitive load and predictability.
Experimental MaterialsWe ran 2 offline and 4 online studies, using different mea-sures on a set of 36 stimuli items. The aim was to empiri-cally establish a solid indication of cognitive load as a func-tion of plausibility and (in part) predictability. The influenceof cloze-probability on cognitive load was reduced by usinglow constraining contexts.
Experimental items All items were German independentmain clauses with uniform syntactic structure (NP-V-ADV-NP). The first NP (sentence subject) was always neutral toavoid an impact on plausibility. The verbs which introducedthe strength of constraint on the subsequent noun were re-strictive (1) or non-restrictive (2). The two different verb cat-egories create constraint and no-constraint contexts to elicitstronger (1) or weaker (2) predictions. The adverb (gleich)following the verb, serves as spill-over region in order to giveparticipants more time to form predictions about the upcom-ing noun. Example: Die Frau bugelt / beschreibt gleich dasT�Shirt / die Socke / den Sessel (see Table 1 for details).
Both of our verb categories were then paired with threedifferent nouns (sentence object) of approximately same fre-quency. Noun frequencies were extracted from word listsDeReWo2 of the German research corpus (DeReKo) and heldconstant for nouns within one item in order to exclude animpact of frequency. Plausibility and predictability were as-sessed by two offline studies described below.
Filler items Fillers were plausible sentences of German,similar to condition (1a). Since a high amount of semanti-cally anomalous sentences could artificially increase the tol-
2(DeReKo corpus size > 28 billion words. Source: Korpusbasierte WortgrundformenlisteDeReWo,2012, v-ww-bll-320000g-2012-12-31-1.0, http://www.ids-mannheim.de/derewo)
erance for implausible sentences, no anomalous fillers wereused. Fillers differed, however, from experimental items inlength and syntactic structure to introduce syntactic variationfor the comprehender. Half of the fillers were followed byyes/no comprehension questions.Analysis The statistical analyses of the data collected fromall experiments were conducted using R statistical program-ming environment (R Core Team, 2013) and the lme4 pack-age (Bates, Maechler, Bolker, & Walker, 2015). Statisticalsignificance was assessed by stepwise reduction of the maxi-mal model’s fixed effects structure.
The Verb and Object fixed factors used for the analysesof all the experiments were contrast coded, thus creating thefollowing factors: O1 which denotes the difference betweenobjects (a) vs. (b); O2 which denotes the difference betweenobjects (b) vs. (c); and their interaction with the Verb, namely,VxO1 for the interaction between the Verb and O1, and VxO2for the interaction between the Verb and O2.
Offline Experiments
First, a plausibility rating determined how plausible on aver-age a noun is in its sentence context. Second, a classical sen-tence completion task for cloze probability norming assessedto what extend the verb constraint would also increase pre-dictability (as in cloze probability) of plausible vs. possiblenouns. Both studies were set up using a web form. Answerswere submitted anonymously via the form or using pen andpaper. We controlled for the unique participation of partici-pants.Plausibility ratingMethod 14 independent German native speakers with ageranging from 18 to 58 years participated in the study volun-tarily without reimbursement. One list contained all our stim-uli sentences in randomized order and without fillers, so thateach participant scored every item once. 216 sentences wereused for our 36 items in 6 conditions (see Table 1). In thetask description, participants were asked to rate the sentencesaccording to their plausibility on a 7�point Likert scale. Thescale ranged from very plausible (1) to not plausible / diffi-cult to imagine (7).Results We substantially changed 3 items which elicitedscores that noticeably deviated from the average rating pat-
pretests
8
So, we conducted:
1. Ling Study
2. Vis Study
9
Results
Results
●
●
●
●
●
●
12
13
14
15
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A −
Valu
es Verbs:●● Audio: iron
Audio: describe
Visual: iron
Visual: describe
●
●
●
●
●
●
12
13
14
15
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A −
Valu
es Verbs:●● Audio: iron
Audio: describe
Visual: iron
Visual: describe
LING
LING
VIS
VIS
ICA
Ling vs. Vis
11
Results
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
ICA
Visual Study
12
Results
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
ICA
Visual Study
13
Results
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
ICA
Visual Study
14
Results
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
ICA
Visual Study
15
Results: Vis Study
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns
Objects:t−shirt
sock
distractor1
distractor2
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns Objects:t−shirt
sock
distractor1
distractor2
face
Referent Gaze Condition
describe sock describe t-shirt iron sock iron t-shirt
Proportion of fixations
16
We saw …
• linguistic + visual context -> concrete predictions
• Verb acts as a pointer to categories of objects
• Situated communication: Gaze cue an inseparable part of the visual context
• Speakers look at entities shortly before mentioning them. (Griffin & Bock, 2000; Meyer et al., 1998)
• Listeners rapidly inspect objects as they are mentioned. (Tanenhaus et al., 1995)
So, we started wondering about … gaze
17
The gaze cue
• Is gaze cue part of the context for the spoken referent?
• If so, how does it influence the surprisal on the referent?
• Is there surprisal on the gaze cue?
18
So, we conducted…
3. Gaze Study
“The woman irons”
“soon the t-shirt.”
no gaze
referent gaze
Gaze conditions:
19
Example trial
Results
●
●
●
●
●
●
●
●
16
17
18
19
20
21
22
23
Subject Verb Adverb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues
Referent gaze.Conditions:●● iron t−shirt
iron sock
describe t−shirt
describe sock
●
●
●
●
●
●
●
●
16
17
18
19
20
21
22
23
Subject Verb Adverb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues
Referent gaze.Conditions:●● iron t−shirt
iron sock
describe t−shirt
describe sock
●●
●
●
●
●
●
●
16
17
18
19
20
21
22
23
Subject Verb Adverb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues
No gaze.Conditions:●● iron t−shirt
iron sock
describe t−shirt
describe sock
●●
●
●
●
●
●
●
16
17
18
19
20
21
22
23
Subject Verb Adverb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues
No gaze.Conditions:●● iron t−shirt
iron sock
describe t−shirt
describe sock
ICA
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
●
●
●●
●
●
16
17
18
19
20
21
22
Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.
Mea
n IC
A va
lues Conditions
●● iron t−shirt
iron sock
describe t−shirt
describe sock
26
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns Objects:t−shirt
sock
distractor1
distractor2
face
No Gaze Condition
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns Objects:t−shirt
sock
distractor1
distractor2
face
Referent Gaze Condition
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns Objects:t−shirt
sock
distractor1
distractor2
face
Referent Gaze Conditiondescribe sock describe t-shirt describe t-shirtdescribe sock
Proportion of fixations
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns
Objects:t−shirt
sock
distractor1
distractor2
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
nsObjects:
t−shirt
sock
distractor1
distractor2
27
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns Objects:t−shirt
sock
distractor1
distractor2
face
No Gaze Condition describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns Objects:t−shirt
sock
distractor1
distractor2
face
Referent Gaze Condition
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns Objects:t−shirt
sock
distractor1
distractor2
face
Referent Gaze Conditiondescribe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns Objects:t−shirt
sock
distractor1
distractor2
face
Referent Gaze Condition
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns Objects:t−shirt
sock
distractor1
distractor2
face
Referent Gaze Condition
iron sock iron t-shirt iron t-shirtiron sock
Proportion of fixations
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
ns
Objects:t−shirt
sock
distractor1
distractor2
describe sock describe t−shirt iron sock iron t−shirt
0.00
0.25
0.50
0.75
1.00
100011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
00
Trial Time (ms)
Prop
ortio
n O
f Fix
atio
nsObjects:
t−shirt
sock
distractor1
distractor2
28
• Visual context concrete predictions
• Surprisal effects @ prediction (dis)confirming
• No cognitive load differences @ prediction forming:
• no active predictions?
• load too low?
• No cognitive load differences @ the gaze cue:
• what if gaze becomes really surprising (incongruent)?
Conclusions
29