Alignment in multimodal dialogue corpora

Robin Hill and Ellen Gurman Bard

Edinburgh

Joint Construction Task

• Collaborative and cooperative joint action in a dyad.

• Two eye-trackers linked in parallel.

• Manipulate communication/feedback channels available from trial to trial:– Speech permitted/denied– Partner’s mouse cursor visible/invisible– Partner’s gaze position visible/invisible

JEL Lab

JCT in action

Cross-recurrence

• Distribution of visual alignment over time

• Contrast with random gaze alignment.

• Richardson, D. C., & Dale, R. (2005). Looking to understand: The coupling between speakers' and listeners' eye movements and its relationship to discourse comprehension. Cognitive Science, 29(6), 1045-1060.

• Richardson, D. C., Dale, R., & Kirkham, N. Z. (2007). The art of conversation is coordination: Common ground and the coupling of eye movements during dialogue. Psychological Science, 18(5), 407-413.

Cross-recurrent Gaze

Richardson & Dale, 2005

In the JCT corpus

• Eye-movement records for the central 120 seconds of each trial used.

• Time-linked overlap = percentage of time with JA gaze-tracks in same Region of Interest for a series of 200ms time lags across a moving 12-second temporal window.

• Baseline = randomly re-ordered series of the same eye-movement record .

• If the two are identical, temporal overlap purely by chance.

Global mean recurrence

24

25

26

27

28

29

30

31

Lag (ms)

Rec

urre

nce

(%)

Dyad recurrenceRandom baseline

•Gaze at dynamic objects well coordinated• Greatest overlap simultaneous•Strongest coincidence approximately in the range ±2000ms

Global mean recurrence

24

25

26

27

28

29

30

31

Lag (ms)

Rec

urre

nce

(%)

Dyad recurrenceRandom baseline

•Symmetrical distribution indicates similar behaviour exhibited by both partners.•Similar to the R&D findings

Joint attention and communication modality

• Coordination is non-random in all conditions including baseline: Joint task always results in some level of gaze coordination.

SpeechMouseGaze

MouseGaze

SpeechGaze

Gaze

SpeechMouse

Mouse

Speech None

21

23

25

27

29

31

33

-6000

-5600

-5200

-4800

-4400

-4000

-3600

-3200

-2800

-2400

-2000

-1600

-1200

-800

-400 0

400

800

1200

1600

2000

2400

2800

3200

3600

4000

4400

4800

5200

5600

6000

Lag (ms)

% R

ecu

rren

ce

Dyad recurrence

Random baseline

21

23

25

27

29

31

33

Lag (ms)%

Re

cu

rre

nc

e

Dyad recurrence

Random baseline

21

23

25

27

29

31

33

Lag (ms)

% R

ecu

rre

nc

e

Dyad recurrence

Random baseline

21

23

25

27

29

31

33

Lag (ms)

% R

ecu

rre

nc

e

Dyad recurrence

Random baseline

21

23

25

27

29

31

33

Lag (ms)

% R

ecu

rre

nc

e

Dyad recurrence

Random baseline

21

23

25

27

29

31

33

Lag (ms)

% R

ecu

rre

nc

e

Dyad recurrence

Random baseline

21

23

25

27

29

31

33

Lag (ms)%

Re

cu

rre

nc

e

Dyad recurrence

Random baseline

21

23

25

27

29

31

33

Lag (ms)

% R

ec

urr

en

ce

Dyad recurrence

Random baseline

Speech summary

• Speech appears to reduce the occurrence of visual alignment.

• Speech tends to increase task completion times.

BUT• People make fewer mistakes during the task when

they can speak.• Speech-improved strategies persist even if dyads are

later prevented from talking to each other.

Next…

• Examine cross-recurrence pinned to specific linguistic elements/events.– E.g. referring expressions.

Eye-voice-eye

• Synchronising of gaze can be broken into two components:Eye-voice span (speaker)+Voice-eye span (speaker-listener)

• Examine the relationship to determine if the communicative burden is slanted towards the speaker of the listener.– E.g. Are there longer production times and do

these lead to shorter comprehension times?

<blink>

Thank you.