blindSight:Eyes-free mobile phone interaction

Kevin Li, University of California, San DiegoPatrick Baudisch, Microsoft ResearchKen Hinckley, Microsoft Research

calendar calendar

preview preview

“Monday 9am”

“tic, tic, sssssh”

“How about Monday morning?”

“Yeah, looks likeI’m free after 10”

blindSight

blindSightis an application running on Microsoft Windows Smartphone

is launched when user places or receive a call. It then replaces the in-call menu

unlike the in-call menu, blindSight uses auditory feedback

why?

PCs…PC screens have the users’ undivided attention

design for the visual channel

eyePhone

environment

visual impairment

screen-less device

can’t seescreen

Lots of information is stored on mobile phones…

… the interfaces are visual

0123456789

# of

Peo

ple

Would Like

Very Important

# o

f pa

rtic

ipa

nts

“I need to access as part of a phone conversation:”

survey

Ok, so let’s just translate all text from visual to auditory

– “Menu: Press 1 to search contacts; press 2 to add a contact; press 3 to access your calendar…”

Wait, that sounds familiar

Please listen carefully as ouroptions have changed…

related work

User’s should be able to “dial ahead” [Perugini et al.,CHI 2007]

Zap and Zoom allows users to jump to locations using shortcuts [Hornstein, UBILAB Rep 1994]

Use visual channel to inform users about options [Yin and Zhai, CHI 2006]

interactive voice response

Time compress audio[Dietz and Yerazunis, UIST 2001]

Integrate speech commands into the conversation [Lyons et al., CHI 2004]

phone interactionmid-conversation

blindSight’sauditory feedback

audio is heard only by the user, not by the person at the other end

rationale

people can recover from audio interruptionsas long as interruption is short

human-human conversation contains redundancy

can we use this redundancy to injectauditory feedback from the device?

how do we make sure device feedback fits into these time windows of low information content?

rules

1. feedback only on-demand

hearvoice note

mute speaker phone

heartask list

addcontact

recordvoice

findcontact

calendar

hearemails

hear textmessage

home

rules

2. brevity

21 3

8

delete

4

97

65

abc def

tuv

ghi

wxyzpqrs

mnojkl

nextplay

find contact

type 6“200 hits”

type 2“12 hits”

type 7“Marion”

rules

3. non-speech previews of composites

week week

day day

3 hours 3 hours

½ hour ½ hourblock½h

previewday

+–––

+

_

whereAmI go today

preview3 hours

++

calendar

(what if the content is a long list,such as appointments for a day?)

rules

4. decomposition

week week

day day

3 hours 3 hours

½ hour ½ hournext

previewday

+–––

+

_

whereAmI go today

preview3 hours

(what if the content is a long list,such as appointments for a day?)

rules

5. interruptability

user interface runs as a separate thread

1 32

64 5

97 8

actiondelete save

rules

6. minimize modes

1 32

64 5

97 8

actiondelete save

mon wedtue

satthu fri

sun

actiondelete save

pick day start time end time

…modes

rules

6. minimize modes (avoid wizards)

week week

day day

3 hours 3 hours

½ hour ½ hourblock½h

previewday

+–––

+

_

whereAmI go today

preview3 hours

++

++

hearvoice note

mute speaker phone

heartask list

addcontact

recordvoice

findcontact

calendar

hearemails

hear textmessage

home

add contact

21 3

8

save

4

97

6

0

5

delete

iterator

1 32

64 5

97 8

action

action

delete save

menu

patterns

calendar

week week

day day

3 hours 3 hours

½ hour ½ hourblock½h

previewday

+––

–

+

_

whereAmI go today

preview3 hours

++

+

+

(shows fast usage by an experienced user)

demo video

hardware

space

epoxy dots

enlarged spaces

0%

2%

4%

6%

8%

10%

12%

14%

16%

1 2 3

Erro

r Rat

e (%

)

Flip

EarThumb

Visual

Flip

Ear

Visual

error

13 2

6 45

9 78

#*

0

.

0%

2%

4%

6%

8%

10%

12%

14%

16%

1 2 3

Erro

r Rat

e (%

)

Flip

EarThumb

Visual

Flip

Ear

Visual

error

blindSightevaluation

interfaces

Smartphone 2003 (sighted)BlindSight (eyes-free) vs.vs.

task

while “driving”idle

(1) schedule appointments and (2) add contacts

0 1 2 3 4 5 6 7 8

Was not missing information

Knew position in the menu

Knew what day/time I was at

Felt in control of the conversation

Better for setting meeting times

Prefer if driving and talking

Prefer Overall

blindSight Smartphone

Overall preference

results

1. brevity is good, but use in moderationclarification of navigation overrides brevity

2. predictable/modeless user interface is key

3. auditory feedback goes a long way even during phone call(disclaimer: need to study how it interferes with activities… driving)

lessons

environment

visual impairment

can’t seescreen

screen-lessdevice

next:

eyePhone

eyePhone

eyesFreePhone

?

blindSight:Eyes-free mobile phone interaction

Kevin Li, University of California, San DiegoPatrick Baudisch, Microsoft ResearchKen Hinckley, Microsoft Research

extra slides

0 1 2 3 4 5 6 7

Could hear partner despite …

Could hear blindSight despite …

Easy to listen and hit buttons

Day preview was useful

3 hour preview was useful

Useful to hear content without …

Tactile dots on keypad were useful

1. built a system

2. a set of eyes-free design rules

3. keypad modifications enabling eyes-free

4. user study comparing with a product (Smartphone 2003)

contributions

1. feedback only on-demand

2. brevity

3. non-speech previews of composites

4. decomposition

5. interruptability

6. minimize modes

rules

iterator

1 32

64 5

97 8

action

action

delete save

menu

patterns

hearvoice note

mute speaker phone

heartask list

addcontact

recordvoice

findcontact

calendar

hearemails

hear textmessage

home

add contact

21 3

8

save

4

97

6

0

5

delete

iterator

1 32

64 5

97 8

action

action

delete save

menu

patterns

calendar

week week

day day

3 hours 3 hours

½ hour ½ hourblock½h

previewday

+––

–

+

_

whereAmI go today

preview3 hours

++

+

+

hearvoice note

mute speaker phone

heartask list

addcontact

recordvoice

findcontact

calendar

hearemails

hear textmessage

21 3

8

save

4

97

6

0

5

delete

week week

day day

3 hours 3 hours

½ hour ½ hourblock½h

previewday

+–––

+

_

whereAmI go today

preview3 hours

++

21 3

8

delete

4

97

65

abc def

tuv

ghi

wxyzpqrs

mnojkl

nextplay

add contact find contact

calendar

home

type type

folder folder

n items n items

item itemplay

+–––

+

_ +preview

+

email, tasks, voice, SMS home

help

hold bottom left for

hold bottom right for

menu

…is a phenomenon in which people who are perceptually blind in a certain area of their visual field demonstrate some visual awareness, without any qualitative experience

blindSight...

[wikipedia]

don’t mode me in

blind sight

10 design rulesto allow eyes-free use and flow

tactile features

…are in in a mobile situation

If they requires visual attention,users will fail at their current activity

interference with social activitiesdrive off the road…

phones…

vs.vs.baseline

interfaces