1

On the Usage of Gaze Informationfor NonCommand Interactive Applications

KHIAT, AbdelazizRobotics Lab.

The 4th COE Technical PresentationJuly 19, 2005

2

Outline

n Introduction on Human Attention and Gaze.

n Gaze tracking techniques and applications:n Technological issues.n Interaction issues.

n A preliminary application:n Implementation.n Observations.

n Gaze behavior during reading tasks.

n Proposed extensions to the application.

%

1000

3

Human Attention or “Interest”

Some factsn Eyes are extremely rapidn Eye movements are natural ⇒ Little conscious effort is needed

n Target acquisition requires to look first before to actingn Direction of gaze implicitly indicates the point of interest

Constrained interface between two powerful information processors

Need to increase bandwidth across the channel

Detecting Human Attention is Possible

%

1000

4

Problems and Issues in Gaze Tracking and Applicationsn Technological issues:

n Tracking methods and their accuracy.n Refinement of noisy data.

n Interaction issues:n Eyes are perceptual devices, not control ones:

Midas touch problem and dwell-time usage.n Need for a new interaction scenario/technique:

Noncommand Interfaces.

n An Open issue:

Can we infer human thoughts from gaze information?

%

1000

5

Technological Issues:vision only based tracking

n Using template matching technique to track the head and

Hough transform to track the cornea

Advantages:n Non-intrusive method and does not

cause discomfort.n Tracks both head pose and gaze

vector

Drawbacks:n Less accurate (±3°)n Need initial setting

[Matsumoto et al. 1998]

%100

0

6

Technological Issues: processing raw gaze data

n Various eye movementsn Convergencen Fixationsn Saccadesn Pursuit motionn Drift and micro-saccades, …

n Refining raw data:n Filter the noise (Butterworth)

n Distinguish Fixationsfrom Saccades

( ) ( ) nc

jH 2

2

11

ωω

ω+

=

[Duchowski 2003]

%

1000

2

7

Technological Issues: saccade/fixation detection

n Velocity-Thresholdn Saccades >300 [deg/sec].n Fixations <100 [deg/sec].n Usual threshold 200 [deg/sec].

n Dispersion-Thresholdn

n Threshold set such that visual angle is between 0.5 [deg] and1[deg].

( ) ( )( ) ( )yy

xxD

minmaxminmax

−−

=

[Duchowski 2003]

%

1000

8

Interaction Issues: using the filtered data

n Command based Interface [Jacob 1993]n Straightforward applications: Objects selection {Menu selection,

Window scrolling, …) ⇒ Pointing.n Midas Touch problem: Eyes are not a control device.n Use dwell time to trigger a selection.

n Non-Command Interfaces [Nielsen 1993]n The computer monitors user’s actions instead of waiting for

his/her command.n Potential Applications: User Support ⇒ Detect difficulties and

provide translation support of difficult words

%

1000

9

Application Overview

Gaze Tracking ModuleUser

Action

Gaze

IE Web

BrowserD

ata buffering &

Feedback

Windows

Linux

Assistance

Module

BHO

%

1000

10

Preliminary Implementation

Pro-Active Dictionary Using Head Direction

x 2

[Khiat et al. 2003]

%

1000

11

Observations

n Continuous consciousness to direct head or gaze

n Unnatural task for human beings.

n Eye movements are not always under voluntary control.

Need information from the user’s natural [eye] movements instead of requiring him/her to make specific ones

%100

0

12

In the Reading Context:how to detect user difficulties?

Gaze pattern during

normal reading

Gaze pattern when

difficulties encountered

Need an Experimental Validation

[Hyrskykari et al. 2003]

%

1000

3

13

Gaze Patterns in Reading:data collection experiment

n Experiment tools:n EMR-NC™ gaze trackern WebTracer recording softwaren MRI-EMR-NC™ data viewing software

n Experiment conditions:n 9 subjects were asked to read 3 texts eachn The subjects had 3 different levels of English proficiencyn The texts were separated into 3 different levels of difficultyn The reading order was done arbitrarily.n Subjects indicate an understanding problem with a mouse click

when this one occurs

EMR-NC

%

1000

14

Gaze Patterns in Reading:data example

Raw

Gaze D

ataP

rocessed Data

Gaze Data Plot Gaze Data Separated in X and Y

%

1000

15

Gaze Patterns in Reading:closer look at a single line

n Observations:n Presence of regressions.

n Variable distance to the moments of the problem.

n Average of regression number is 1.3(for the 40 cases analyzed).

n Sometimes no regression is noticed before the problem occurs (10% of the cases)

Regressions can be used to detect problem’s occurrence

[Khiat et al. 2004]

%

1000

16

Proposed context grounding

n Gaze information not enough all alonen Propose a grounding with a context [indicated by gaze]

n Implicit preprocessing of the viewed scene (text).n Associate a difficulty rate to each word.n Difficulty rate based on the frequency of usage.

n Preprocessing step can be also used for:Text parsing, conversion to basic form, link appropriate reading, …and so on.

%

1000

17

References

n Duchowski, A.T. “Eye Tracking Methodology: Theory and Practice”, Springer-Verlag, 2003.

n Hyrskykari et al. “Proactive Response to Eye Movements”, in Proceedings of the IFIP INTERACT 2003.

n Jacob, R.K. “Eye Movement-based Human-Computer Interaction Techniques”, in Advances in Human-Computer Interaction, pp.150-190, Vol. 4, 1993.

n Khiat et al. “Towards Gaze-based Proactive Support for Web Readers”, in Proceedings of the IEEE RO-MAN 2003.

n Khiat et al. “Task Specific Eye Movements Understanding for a Gaze-Sensitive Dictionary”, in Proceedings of the ACM IUI 2004.

n Matsumoto et al. “An Algorithm for Real-Time Stereo Vision Implementation of Head Pose and Gaze Direction Measurement”, in Proceedings of the IEEE FG’2000.

n Nielsen J. “NonCommand User Interfaces”, in Communications of the ACM, pp. 83-99, 36(4), 1993.

%100

0

18

Thank you

%

1000