Slide 1

blindSight: Eyes-free mobile phone interaction Kevin Li, University of California, San Diego Patrick Baudisch, Microsoft Research Ken Hinckley, Microsoft Research Slide 2 calendar preview Monday 9am tic, tic, sssssh How about Monday morning? Yeah, looks like Im free after 10 blindSight Slide 3 is an application running on Microsoft Windows Smartphone is launched when user places or receive a call. It then replaces the in-call menu unlike the in-call menu, blindSight uses auditory feedback Slide 4 why? Slide 5 PCs PC screens have the users undivided attention design for the visual channel Slide 6 eyePhone Slide 7 environment Slide 8 visual impairment Slide 9 screen-less device Slide 10 cant see screen Slide 11 Slide 12 Slide 13 Lots of information is stored on mobile phones Slide 14 the interfaces are visual Slide 15 # of participants I need to access as part of a phone conversation: survey Slide 16 Ok, so lets just translate all text from visual to auditory Menu: Press 1 to search contacts; press 2 to add a contact; press 3 to access your calendar Wait, that sounds familiar Slide 17 Please listen carefully as our options have changed Slide 18 related work Slide 19 Users should be able to dial ahead [Perugini et al.,CHI 2007] Zap and Zoom allows users to jump to locations using shortcuts [Hornstein, UBILAB Rep 1994] Use visual channel to inform users about options [Yin and Zhai, CHI 2006] interactive voice response Slide 20 Time compress audio [Dietz and Yerazunis, UIST 2001] Integrate speech commands into the conversation [Lyons et al., CHI 2004] phone interaction mid-conversation Slide 21 blindSights auditory feedback Slide 22 audio is heard only by the user, not by the person at the other end Slide 23 rationale people can recover from audio interruptions as long as interruption is short human-human conversation contains redundancy can we use this redundancy to inject auditory feedback from the device? Slide 24 how do we make sure device feedback fits into these time windows of low information content? Slide 25 rules 1. feedback only on-demand hear voice note mute speaker phone hear task list add contact record voice find contact cal endar hear emails hear text message home Slide 26 Slide 27 Slide 28 rules 2. brevity 2 13 8 delete 4 9 7 6 5 abc def tuv ghi wxyz pqrs mno jkl next play find contact type 6 200 hits type 2 12 hits type 7 Marion Slide 29 rules 3. non-speech previews of composites week day 3 hours hour block h preview day + + _ whereAmI go today preview 3 hours ++ calendar (what if the content is a long list, such as appointments for a day?) Slide 30 Slide 31 Slide 32 rules 4. decomposition week day 3 hours hour next preview day + + _ whereAmI go today preview 3 hours (what if the content is a long list, such as appointments for a day?) Slide 33 rules 5. interruptability user interface runs as a separate thread Slide 34 1 3 2 6 4 5 9 7 8 action deletesave rules 6. minimize modes 1 3 2 6 4 5 9 7 8 action deletesave mon wed tue sat thu fri sun action deletesave pick daystart timeend time modes Slide 35 rules 6. minimize modes ( avoid wizards) week day 3 hours hour block h preview day + + _ whereAmI go today preview 3 hours ++ + + Slide 36 hear voice note mute speaker phone hear task list add contact record voice find contact cal endar hear emails hear text message home Slide 37 add contact 2 1 3 8 save 4 9 7 6 0 5 delete Slide 38 iterator 1 3 2 6 4 5 9 7 8 action deletesave menu patterns Slide 39 calendar week day 3 hours hour block h preview day + + _ whereAmI go today preview 3 hours ++ + + Slide 40 (shows fast usage by an experienced user) demo video Slide 41 Slide 42 Slide 43 hardware Slide 44 Slide 45 Slide 46 Slide 47 space Slide 48 epoxy dots enlarged spaces Slide 49 Flip Ear Visual error Slide 50 Slide 51 1 3 2 6 4 5 9 7 8 # * 0. Slide 52 Flip Ear Visual error Slide 53 blindSight evaluation Slide 54 interfaces Smartphone 2003 (sighted)BlindSight (eyes-free) vs. Slide 55 task while drivingidle (1) schedule appointments and (2) add contacts Slide 56 012345678 Was not missing information Knew position in the menu Knew what day/time I was at Felt in control of the conversation Better for setting meeting times Prefer if driving and talking Prefer Overall blindSightSmartphone Overall preference results Slide 57 1. brevity is good, but use in moderation clarification of navigation overrides brevity 2. predictable/modeless user interface is key 3. auditory feedback goes a long way even during phone call (disclaimer: need to study how it interferes with activities driving) lessons Slide 58 environment visual impairment cant see screen screen-less device next: Slide 59 eyePhone Slide 60 eyesFreePhone ? Slide 61 blindSight: Eyes-free mobile phone interaction Kevin Li, University of California, San Diego Patrick Baudisch, Microsoft Research Ken Hinckley, Microsoft Research Slide 62 Slide 63 extra slides Slide 64 Slide 65 1. built a system 2. a set of eyes-free design rules 3. keypad modifications enabling eyes-free 4. user study comparing with a product (Smartphone 2003) contributions Slide 66 1. feedback only on-demand 2. brevity 3. non-speech previews of composites 4. decomposition 5. interruptability 6. minimize modes rules Slide 67 iterator 1 3 2 6 4 5 9 7 8 action deletesave menu patterns Slide 68 hear voice note mute speaker phone hear task list add contact record voice find contact cal endar hear emails hear text message home Slide 69 add contact 2 1 3 8 save 4 9 7 6 0 5 delete Slide 70 iterator 1 3 2 6 4 5 9 7 8 action deletesave menu patterns Slide 71 calendar week day 3 hours hour block h preview day + + _ whereAmI go today preview 3 hours ++ + + Slide 72 hear voice note mute speaker phone hear task list add contact record voice find contact cal endar hear emails hear text message 2 1 3 8 save 4 9 7 6 0 5 delete week day 3 hours hour block h preview day + + _ whereAmI go today preview 3 hours ++ 2 13 8 delete 4 9 7 6 5 abc def tuv ghi wxyz pqrs mno jkl next play add contact find contact calendar home type folder n items item play + + _ + preview + email, tasks, voice, SMS home help hold bottom left for hold bottom right for menu Slide 73 is a phenomenon in which people who are perceptually blind in a certain area of their visual field demonstrate some visual awareness, without any qualitative experience blindSight... [wikipedia] Slide 74 dont mode me in blind sight 10 design rules to allow eyes-free use and flow tactile features Slide 75 are in in a mobile situation If they requires visual attention, users will fail at their current activity interference with social activities drive off the road phones Slide 76 Slide 77 vs. baseline interfaces Slide 78 task times