Towards Gaze-Based Wayfinding Assistancewp.geogaze.org/wp-content/uploads/2016/02/Towards... ·...

||

Peter KieferColloquium, Center of Cognitive Science, University of Freiburg

18 June 2015

18 Jun 2015 1

Towards Gaze-Based Wayfinding Assistance

Peter Kiefer

||

Peter Kiefer Researcher, Lecturer

(«Oberassistent I») PhD in Applied Computer

Science University of Bamberg (2011) «Mobile Intention Recognition» with Prof. Christoph Schlieder

Martin Raubal Chair of Geo Information

Engineering

18 Jun 2015Peter Kiefer 2

GeoGazeLab

Ioannis Giannopoulos,David Rudi PhD Students Computer Scientists

|| 18 Jun 2015Peter Kiefer 3

Aviation SafetyGeo HCIWayfinding

Cognitive processesand visual attentionduring pedestrianwayfinding.

Gaze-basedinteraction withgeographicinformation

Pilot’s visual searchstrategies in thecockpit

Gaze-BasedWayfinding Assistance

PhD DavidRudi

LAMETTAProject with

SWISSPhD IoannisGiannopoulos

PhD N.N.(start Sep 15)

http://www.geogaze.org/



Example: GazeNav

Giannopoulos, I., Kiefer, P., & Raubal, M. (2015)GazeNav: Gaze-based pedestrian navigation.In: 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI). ACM: New York, accepted


Example: GazeNav

Mobile HCI 2015

Gaze point


Example: GazeNav

Mobile HCI 2015

||

Wayfinding and Visual Attention

Gaze-Based Interaction with Maps

Gaze-Based Wayfinding Assistance


||

Measuring visual attention Application domains

Psychology, consumer research, economy, architecture, arts, design,pilot safety training, …

18 Jun 2015Kiefer, Giannopoulos, Raubal 9

Eye Tracking

Sour

ces:

Sta

tionä

rer

Eye

Tra

cker

: Sen

som

otor

ic In

stru

men

ts (

http

://w

ww

.sm

ivis

ion.

com

/)K

unst

von

Lau

ra W

alde

, Gaz

etra

ck: P

eter

Kie

fer,

Rec

hts:

Geo

info

rmat

ions

-Eng

inee

ring

“Remote“Eye Tracking

MobileEye

Tracking

http://www.smivision.com/


Eye tracking on the mass market?

http

://w

ww

.blo

gcdn

.com

/ww

w.e

ngad

get.c

om/m

edia

/201

3/03

/dsc

0787

3-13

6329

4641

.jpg

http

s://t

heey

etrib

e.co

m/p

rodu

cts/

http

://w

ww

.tech

rada

r.com

/new

s/po

rtabl

e-de

vice

s/go

ogle

-pat

ents

-eye

-trac

king

-for-

goog

le-g

lass

-109

1428

http

://bl

og.la

ptop

mag

.com

/firs

t-ultr

aboo

k-ey

e-tra

ckin

g

http://www.blogcdn.com/www.engadget.com/media/2013/03/dsc07873-1363294641.jpg

http://www.techradar.com/news/portable-devices/google-patents-eye-tracking-for-google-glass-1091428

http://www.techradar.com/news/portable-devices/google-patents-eye-tracking-for-google-glass-1091428

http://blog.laptopmag.com/first-ultrabook-eye-tracking


What about privacy?

http://comm

ons.wikim

edia.org/wiki/File:G

oogle_Glass_detail.jpg

GN

U Free D

ocumentation License, V

ersion 1.2, Author: A

ntonio Zugaldia

(C) Google, http://www.youtube.com/watch?v=9c6W4CCU9M4

http://commons.wikimedia.org/wiki/File:Google_Glass_detail.jpg

http://www.youtube.com/watch?v=9c6W4CCU9M4

||

Navigation = Locomotion + Wayfinding (Montello, D.R., 2005)


Wayfinding

Body movementAvoiding obstacles

Goal directed movementPlanningDecision making

||

Example (Downs&Stea, 1977)


Cognitive Processes in Wayfinding

Orientation Route decision

Keeping on track

Recognizingdestination

||

Goal Intelligent user interfaces Automated assistance based on user’s cognitive states

Empirical studies Model of user activities and cognitive processes

Engineering Recognizer using the model Adaptive system


Cognitive Engineering

||

Self-Localization «Please mark your position

on this map» Map symbols and

corresponding landmarks Some landmarks visible

Requires visual search and logical inference Eye tracking measures only

search


A wayfinding study

Kiefer, P., Giannopoulos, I., & Raubal, M. (2014). Where am I? Investigating map matching during self‐localization with mobile eye tracking in an urban environment. Transactions in GIS, 18(5), 660-686.

||

Self-Localization «Please mark your position

on this map» Map symbols and

corresponding landmarks Some landmarks visible

Requires visual search and logical inference Eye tracking measures only

search


A wayfinding study

TGIS 2014


Self-localization

TGIS 2014

||

RQ1 Do successful participants

spend more visual attention on map symbols that have a visible corresponding landmark than unsuccessful participants?

(A distribution measure.)

RQ2 Do successful participants have

more switches of visual attention between symbols on the map and their corresponding landmarks in the environment?

(A sequence measure.)


Research Questions

TGIS 2014

||

Successful participants spent significantly more time fixating AOIs from the helpful AOIs category than the unsuccessful participants

Successful participants had significantly more switches of visual attention between symbols on the map and their corresponding landmarks in the environment than did unsuccessful participants.


Results

TGIS 2014

||

LA

A relA

E

LErelE

S

LS

relS

E: the church over thereA: the church XS: the church icon

L : the location right of the churchE

L : the location behind the churchA

L : the location North of the churchS


Reference system transformation

Kiefer, P., Scheider, S., Giannopoulos, I., & Weiser, Paul (2015) A wayfinding grammar based on referencesystem transformations. In: Conference on SpatialInformation Theory XII (COSIT). Springer. accepted.


A Wayfinding Grammar

COSIT 2015

Self-localization

Planning

Instruction execution

Instruction execution

||


Gaze-Based Interaction with MapsGaze-Based InteractionGeoGazemarksGaze Map MatchingActivity Recognition

Gaze-Based Wayfinding Assistance


||

Gaze as input modality Real-time gaze processing Intelligent assistance Midas touch problem


Gaze-Based Interaction

Eye typing (Majaranta et al., 2006) Zooming/panning (Stellmach, Dachselt, 2012)

A) Explicit Interaction«What you look at is whatyou get»

||

B) Implicit Interaction Assistance in the background User does not intend to

trigger an action


Gaze-Based Interaction (2)

Activity recognition (Bulling et al., 2011)Subtle Gaze Direction (Bailey et al., 2009)


How can we use gaze-based interactionfor geographic information?

Interacting with a map• GeoGazemarks• Gaze Map Matching• Activity Recognition

Interacting with the environment• GazeNav• Tourist Recommender (future)

Interacting with both• A combined wayfinding

assistant (future)


GeoGazemarks

Giannopoulos, I., Kiefer, P., & Raubal, M. (2012). GeoGazemarks: Providing gaze history for the orientation on small display maps. In Proceedings of the 14th ACM international conference on Multimodal interaction (pp. 165-172). ACM.

||

Content-independent e.g., number of fixations, speed of

saccades, average saccade length, scan path length, etc.

Content-dependent= AOI-based e.g., time to first AOI hit, number of

AOI-returns,%of fixations in an AOI,dwell time, AOI-sequence similarities, etc.

Example: «Which percentage of fixations was spent on the legend of the map?»


Analyses of eye tracking data

Çöl

teki

net

al.

(200

9)

Kiefer, P., & Giannopoulos, I. (2012).Gaze map matching: mapping eye tracking data to geographic vector features.In: Proc. of the 20th International Conference on Advances in Geographic Information Systems (pp. 359-368). ACM.

||

Drawbacks of AOIs Potentially ambiguous Inaccuracy Manual definition

Idea: Gaze Map Matching Digital cartography:

The content is available! Towards an automated

interpretation w.r.t vectorfeatures


Gaze Map Matching

ACM GIS 2012


Example dataset

ACM GIS 2012

||

Example: Hidden Markov Model


ACM GIS 2012


Visual attention depends on the task

Imag

e: h

ttp://

com

mon

s.w

ikim

edia

.org

/wik

i/File

:Yar

bus_

The_

Vis

itor.j

pg/(

publ

ic d

omai

n)

Yarbus (1967)

http://commons.wikimedia.org/wiki/File:Yarbus_The_Visitor.jpg/

||

«Eye movements depend on task (activity)» Infer activity from eye movement measures?


Gaze-Based Activity Recognition

Gaze-based activity recognitionfor office activities

(Bulling et al., 2011)

||

6 Activities Free exploration (6 stimuli) Global search (9) Route planning (8) Focused search (5) Line following (8) Polygon comparison (4)

Random order of stimuli Google Maps 17 participants (no experts) 587 valid recordings, each cut

after 20 seconds


Data Collection

Data collection setup

Kiefer, P., Giannopoulos, I., & Raubal, M. (2013). Using eye movements to recognize activities on cartographic maps. In Proc of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (pp. 478-481). ACM.


Do you see Mickhausen (top right) and Boos (bottom left)?Please, search for the shortest route from Mickhausen to Boos.

ACM GIS 2013


After a moment: «Please, fixate now on Mickhausen.The task begins in 3, 2, 1 ...»

ACM GIS 2013


ACM GIS 2013

||

229 features based on Blinks (5) Fixations (17) Saccades (11) Saccadic direction (100) Direction sequences (96)

Leading to 587 instances of [f1, ... f229, true-activity] Training set (90%) Test set (10%)

[..., true-activity, assigned-activity]

Support Vector Machine (SVM) 10-fold cross validation


Machine Learning

ACM GIS 2013


Results

true activity precision(%)

pred

icte

d ac

tivity

1 2 3 4 5 61 82 10 3 11 2 0 75.92 14 112 5 18 3 0 73.73 1 2 65 1 19 0 73.94 4 7 5 54 1 0 76.15 1 0 21 1 76 1 76.06 0 0 1 0 0 67 98.5∑ 102 131 100 85 101 68

recall (%) 80.4 85.5 65.0 63.5 75.3 98.5accuracy = 77.7%

1: free exploration, 2: search, 3: route planning,4: focused search, 5: line following, 6: polygon comparison.

ACM GIS 2013


Outlook: Intention-awaregaze-based assistance on maps

Kiefer, P. (2015)Blickbasierte Mensch-Computer-Interaktion mit Geoinformationssystemen.In: Thomas H. Kolbe, Ralf Bill, and Andreas Donaubauer, editors, Geoinformationssysteme 2015,pp. 47-58. Wichmann: Heidelberg

||


Gaze-Based Interaction with Maps

Gaze-Based Wayfinding Assistance Technological challengesGazeNav



How can we use gaze-based interactionfor geographic information?

Interacting with a map• GeoGazemarks• Gaze Map Matching• Activity Recognition

Interacting with the environment• GazeNav• Tourist Recommender (future)

Interacting with both• A combined wayfinding

assistant (future)

||

Sunlight ... interferes with infrared Dikablis Saves two videos, manual

post-processing frame-by-frame

Labour-intensive! SMI Glasses Sunshades


Technological Challenges (1)

||

Determining Objectof Interest Dikablis Marker-based solution Labour-intensive!

MSc theses Pius Mosimann (2013) Simple head-tracking helmet 3D city model



||



Simon Haesler (2014) Web service for the 3D intersection



||



Simon Haesler (2014) Web service for the 3D intersection

Yufan Miao (ongoing) Image-based localization




A cognition-aware wayfinding assistant

It seems like you are

confused

The landmark I was talking about is the HCL building. It is located left of HCI.

The building you are

looking at is the HCI building.


GazeNav: Gaze-Based Pedestrian Navigation

Mobile HCI 2015

Gaze point


GazeNav: User Study in a Virtual Environment

Mobile HCI 2015

||

Between subject GazeNav vs. turn-

by-turn 32 participants (13f) 16 per condition


GazeNav: User Experience (UEQ)

Mobile HCI 2015


Summary

Geo HCIWayfinding

Cognitive processesand visual attentionduring pedestrianwayfinding.

Gaze-basedinteraction withgeographicinformation

TGIS 2014Self-localization

Only major papers listed. Complete list at http://www.geogaze.org/

GIScience 2014Complexity of

decision situations

COSIT 2015Wayfinding Grammar

ICMI 2012GeoGazemarks

ACM GIS 2012Gaze Map Matching

ACM GIS 2013Activity Recognition

Gaze-BasedWayfinding Assistance

MobileHCI 2015GazeNav



Do we even need cognition-aware wayfindingassistants?

Pfeiffer, M., Dünte, T., Schneegass, S., Alt, F., & Rohs, M. (2015).Cruise Control for Pedestrians: Controlling Walking Direction using Electrical MuscleStimulation. Proc. of CHI’15

http

s://w

ww.

yout

ube.

com

/wat

ch?v

=JS

fnm

_HoU

v4

||

January 17-22, 2016 Monte Verità, Ticino, CH For PhD students and early

PostDocs who are usingeye tracking intheir research

Lectures Hands-on sessions Student presentations


ETH Zurich Winter SchoolEye Tracking – Experimental Design, Implementation, and Analysis

||

Keynote Speaker Anke Huckauf General Psychology,

Ulm University (DE)

Lecturers Andrew Duchowski Computer Science,

Clemson University (S.C., USA) Scott MacKenzie HCI, York University (CA)

Izabela Krejtz Applied Cognitive Studies,

University of Social Science and Humanities, Warsaw (PL)

Krzysztof Krejtz Interactive Technologies Lab,

University of Social Science and Humanities, Warsaw (PL)

53

Speakers

18 Jun 2015Peter Kiefer


Thank you for your attention!

Questions and Discussion

http://www.geogaze.org

http://winterschool.ethz.ch/


http://winterschool.ethz.ch/

Towards Gaze-Based Wayfinding Assistancewp.geogaze.org/wp-content/uploads/2016/02/Towards... ·...

Documents

Transcript of Towards Gaze-Based Wayfinding Assistancewp.geogaze.org/wp-content/uploads/2016/02/Towards... ·...