BAM! Brain-Compatible Learning Strategies That Work! Brad Brinkley, Leslie Brinkley, Lee Ann Smith.
Mobile Devices for Control of Ubiquitous Multimedia Brad A. Myers [email protected] Brad A. Myers...
-
date post
20-Dec-2015 -
Category
Documents
-
view
220 -
download
0
Transcript of Mobile Devices for Control of Ubiquitous Multimedia Brad A. Myers [email protected] Brad A. Myers...
Mobile Devices for Control ofUbiquitous Multimedia
Mobile Devices for Control ofUbiquitous Multimedia
Brad A. Myers
http://www.pebbles.hcii.cmu.edu/
Brad A. Myers
http://www.pebbles.hcii.cmu.edu/
Human Computer Interaction Institute Human Computer Interaction Institute
School of Computer ScienceSchool of Computer Science
Carnegie Mellon UniversityCarnegie Mellon University
Brad Myers Carnegie Mellon 2
Handheld Devices Typically, handheld devices for:
Communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement
“PocketPC” As a media player
Handheld Devices for Remote Control of other devices and media Remote Control of PCs Remote Control of appliances
Typically, handheld devices for: Communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement
“PocketPC” As a media player
Handheld Devices for Remote Control of other devices and media Remote Control of PCs Remote Control of appliances
Brad Myers Carnegie Mellon 6
Approach
Use a personal handheld device as an Interface to the PC and to the Appliances
Use a personal handheld device as an Interface to the PC and to the Appliances
Brad Myers Carnegie Mellon 7
History of Communication
ParcTab – continuous communication
Early “organizers” – no communication
Newton – extra cost for synchronization
Palm – HotSync once a day
WindowsCE/PocketPC –ActiveSync once a day
ParcTab – continuous communication
Early “organizers” – no communication
Newton – extra cost for synchronization
Palm – HotSync once a day
WindowsCE/PocketPC –ActiveSync once a day
Brad Myers Carnegie Mellon 8
Now, Handhelds are Communicating
802.11
BlueTooth
Cell-phone network (G3)
802.11
BlueTooth
Cell-phone network (G3)
Toshiba e740
HP iPaq 3870
Microsoft SmartPhone
Brad Myers Carnegie Mellon 9
Pebbles Project
Performed as part of the Pebbles Project Overall goal: Use of multiple devices at the
same time Palm and desktop PC Mobile Phone and “Smart Home” Multiple handhelds in a meeting Pocket PC and appliances
Multiple users with their devices Single user with multiple devices
Performed as part of the Pebbles Project Overall goal: Use of multiple devices at the
same time Palm and desktop PC Mobile Phone and “Smart Home” Multiple handhelds in a meeting Pocket PC and appliances
Multiple users with their devices Single user with multiple devices
Brad Myers Carnegie Mellon 10
Pebbles is:
PEBBLES
PEBBLES
DAs forDAs forntry ofntry ofothothytes andytes andocations fromocations fromxternalxternalources.ources.
DAs forDAs forntry ofntry ofothothytes andytes andocations fromocations fromxternalxternalources.ources.
http://www.pebbles.hcii.edu/http://www.pebbles.hcii.edu/
Brad Myers Carnegie Mellon 11
Pebbles for MultiMedia
How will handheld devices fit into the user’s overall media space? Where does media come from? Where is the media is displayed? How is the media controlled?
How can handheld devices improve the user interfaces of other media devices, rather than just being another gadget to master?
How will handheld devices fit into the user’s overall media space? Where does media come from? Where is the media is displayed? How is the media controlled?
How can handheld devices improve the user interfaces of other media devices, rather than just being another gadget to master?
Controlling a PCControlling a PC
Remote Commander Slide Show Commander Command Post of the Future Shortcutter
Remote Commander Slide Show Commander Command Post of the Future Shortcutter
http://www.pebbles.hcii.edu/http://www.pebbles.hcii.edu/
Brad Myers Carnegie Mellon 13
Remote Commander
Allow PDAs to control a PC Can be used with any application Uses the standard (single) cursor Don’t have to jump up and
grab mouse All mouse and keyboard functions
Use PDA like touchpad Graffiti or our own pop-up
keyboard Word prediction and completion
Allow PDAs to control a PC Can be used with any application Uses the standard (single) cursor Don’t have to jump up and
grab mouse All mouse and keyboard functions
Use PDA like touchpad Graffiti or our own pop-up
keyboard Word prediction and completion
Brad Myers Carnegie Mellon 14
PocketPC version Get PC’s screen onto PocketPC Full view, or one-to-one zooming
Scroll with iPaq’s buttons Control or scribble
Get PC’s screen onto PocketPC Full view, or one-to-one zooming
Scroll with iPaq’s buttons Control or scribble
Brad Myers Carnegie Mellon 15
For People with Muscular Disabilities
Using handhelds as interface to PCs People with Muscular Dystrophy
have fine-motor control but losegross motor control
Difficulties with mouse and keyboard,but stylus OK
So use Remote Commanderas PC’s keyboard and mouse
Using handhelds as interface to PCs People with Muscular Dystrophy
have fine-motor control but losegross motor control
Difficulties with mouse and keyboard,but stylus OK
So use Remote Commanderas PC’s keyboard and mouse
Brad Myers Carnegie Mellon 16
Alphabet for Accurate Gestures Use a square hole in a template to increase
accuracy Just as easy to learn
as Graffiti™ 18% more accurate
for able novices (p<.02)
Patent Pending Also works with joystick
Use a square hole in a template to increase accuracy Just as easy to learn
as Graffiti™ 18% more accurate
for able novices (p<.02)
Patent Pending Also works with joystick
Brad Myers Carnegie Mellon 17
Power Point Control Use PC to give the presentation Use hand-held to control the PC
Two-way communication Hand-held shows picture of slide,
notes, timer
Use PC to give the presentation Use hand-held to control the PC
Two-way communication Hand-held shows picture of slide,
notes, timer
Brad Myers Carnegie Mellon 18
SlideShow Commander, cont.
See list of slide titles Scribble Tap on on-screen buttons
and hyper-links Preview other
slides Control
demonstrations
See list of slide titles Scribble Tap on on-screen buttons
and hyper-links Preview other
slides Control
demonstrations
Brad Myers Carnegie Mellon 19
Control at a Distance
Controlling a PC fromacross the room Meetings, etc.
Use a laser pointer? Studies
Controlling a PC fromacross the room Meetings, etc.
Use a laser pointer? Studies
Brad Myers Carnegie Mellon 20
Command Post of the Future Share media & control across multiple devices
Big displays, speech, gesture And handhelds
Share media & control across multiple devices Big displays, speech, gesture And handhelds
Brad Myers Carnegie Mellon 21
CPOF, cont.
Provides remote control of main PC battleboard from handhelds
Also, private drill-down of public information Small versions of map and data
visualizations on handhelds Interact locally, and display
changes and annotationswhen ready
Provides remote control of main PC battleboard from handhelds
Also, private drill-down of public information Small versions of map and data
visualizations on handhelds Interact locally, and display
changes and annotationswhen ready
Brad Myers Carnegie Mellon 22
Shortcutter
User-created panels of controls Create custom interfaces and
extensions to PC applications And then take them with you
Direct manipulationfor edit, then setproperties
PocketPC or Palm
User-created panels of controls Create custom interfaces and
extensions to PC applications And then take them with you
Direct manipulationfor edit, then setproperties
PocketPC or Palm
Brad Myers Carnegie Mellon 23
Shortcutter Widgets
Buttons Virtual Joy stick Virtual Knob Sliders Mouse pad Gesture panel
Buttons Virtual Joy stick Virtual Knob Sliders Mouse pad Gesture panel
Brad Myers Carnegie Mellon 24
Scenarios of Use
Lean-back mail reading
Controlling media players
… and many others
Lean-back mail reading
Controlling media players
… and many others
Controlling AppliancesControlling Appliances
PhD research of Jeffrey Nichols Assisted by undergrads and MS students
PhD research of Jeffrey Nichols Assisted by undergrads and MS students
http://www.pebbles.hcii.edu/puc/http://www.pebbles.hcii.edu/puc/
Problem
Too many complex devices, eachwith its own idiosyncratic interface Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock
Increasingly computerized Low usability
Too many complex devices, eachwith its own idiosyncratic interface Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock
Increasingly computerized Low usability
Brad Myers Carnegie Mellon 27
Existing “universal” controls
Pre-programmed at the factory with a subset, Or, laboriously hand-programmed by the user
Pre-programmed at the factory with a subset, Or, laboriously hand-programmed by the user
Brad Myers Carnegie Mellon 28
Our Approach Handheld is a “Personal Universal Controller” (PUC) Two-way communication
Appliance describes its functions Handheld PUC
Automatically creates interface Controls the appliance Displays feedback about appliance status
Handheld is a “Personal Universal Controller” (PUC) Two-way communication
Appliance describes its functions Handheld PUC
Automatically creates interface Controls the appliance Displays feedback about appliance status
Specifications
Control
Feedback
Brad Myers Carnegie Mellon 29
Automatic Generation of UIs
Benefits All interfaces consistent for a user
With conventions of the handheld Even from multiple manufacturers
Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI)
A Hard Problem Previous automatic systems have not
generated high quality interfaces
Benefits All interfaces consistent for a user
With conventions of the handheld Even from multiple manufacturers
Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI)
A Hard Problem Previous automatic systems have not
generated high quality interfaces
Brad Myers Carnegie Mellon 30
Research Challenges
1. Automatic Design of Graphical User Interfaces
2. Automatic Design of Speech User Interfaces
3. Connection with real devices Through various protocols X-10, AV/C, HAVi, UPnP, etc. Also, custom hardware and software
1. Automatic Design of Graphical User Interfaces
2. Automatic Design of Speech User Interfaces
3. Connection with real devices Through various protocols X-10, AV/C, HAVi, UPnP, etc. Also, custom hardware and software
Brad Myers Carnegie Mellon 31
Hand-Generated Graphical Interfaces
First, Hand-Designed PocketPC interfaces: AIWA Shelf Stereo (Tape,CD,Tuner) AT&T Telephone/Answering
Machine
Used Embedded Visual Basic
Ensured quality with heuristicanalysis and think-aloud studies
Compared with manufacturer’sinterfaces
First, Hand-Designed PocketPC interfaces: AIWA Shelf Stereo (Tape,CD,Tuner) AT&T Telephone/Answering
Machine
Used Embedded Visual Basic
Ensured quality with heuristicanalysis and think-aloud studies
Compared with manufacturer’sinterfaces
Brad Myers Carnegie Mellon 32
Results of Comparison
Using PUC, users took 50% less time & made 50% fewer errors
All differences are significant (p < 0.05)
Using PUC, users took 50% less time & made 50% fewer errors
All differences are significant (p < 0.05)
Brad Myers Carnegie Mellon 33
Current PUC Specification Language
XML Full documentation for the
specification language and protocol:
http://www.pebbles.hcii.cmu.edu/puc/
Contains sample specification for a stereo
XML Full documentation for the
specification language and protocol:
http://www.pebbles.hcii.cmu.edu/puc/
Contains sample specification for a stereo
Brad Myers Carnegie Mellon 34
Properties of PUC Language
State variables & commands Each can have multiple labels
Useful when not enough room Typed variables
Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc.
Optional labels for values Hierarchical Structure
Groups
State variables & commands Each can have multiple labels
Useful when not enough room Typed variables
Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc.
Optional labels for values Hierarchical Structure
Groups
Brad Myers Carnegie Mellon 35
Dependency Information
Crucial for high-quality interfaces Expressed as <active-if> clauses
Operations: Equals, Less-Than,
Greater-Than Combined Logically
AND, OR Used for:
Dynamic graying out Layout Widget selection
Crucial for high-quality interfaces Expressed as <active-if> clauses
Operations: Equals, Less-Than,
Greater-Than Combined Logically
AND, OR Used for:
Dynamic graying out Layout Widget selection
Brad Myers Carnegie Mellon 36
Specifications
Have working specifications for: Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator Parts of GMC Yukon Denali SUV Etc.
Have working specifications for: Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator Parts of GMC Yukon Denali SUV Etc.
Brad Myers Carnegie Mellon 37
Controller Generators
iPaq PocketPC
SmartPhone No touchscreen
Desktop (TabletPC)
Speech
iPaq PocketPC
SmartPhone No touchscreen
Desktop (TabletPC)
Speech
Brad Myers Carnegie Mellon 38
Examples of Generated GUIs
Shelf stereo Windows Media Player
Shelf stereo Windows Media Player
Brad Myers Carnegie Mellon 39
Generating Speech Interfaces “Universal Speech Interface” (USI) project
Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi
Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with disambiguation and explanation
Supports queries and spoken feedback Paraphrases as confirmation
“Universal Speech Interface” (USI) project Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi
Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with disambiguation and explanation
Supports queries and spoken feedback Paraphrases as confirmation
Brad Myers Carnegie Mellon 40
Adaptors
“Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly
Custom hardware Custom software
Lutron Windows Media Player
X-10 Light switches, etc.
“Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly
Custom hardware Custom software
Lutron Windows Media Player
X-10 Light switches, etc.
AV/C (standard protocol) Sony CamCorder
HAVi UPnP
Axis Camera
AV/C (standard protocol) Sony CamCorder
HAVi UPnP
Axis Camera
Brad Myers Carnegie Mellon 41
Demonstration
Sony CamCorder Windows Media Player Image library Two way communication:
When state changes from application or appliance, GUI is updated
Can query state with speech
Sony CamCorder Windows Media Player Image library Two way communication:
When state changes from application or appliance, GUI is updated
Can query state with speech
Brad Myers Carnegie Mellon 42
Current Work on PUC Controlling User “Experiences”
One command for multiple devices Play a DVD sets DVD, cable, TV Setting alarm clock sets heat, coffee, car
Consistency for Users And user preferences
More devices UPnP Etc.
Controlling User “Experiences” One command for multiple devices Play a DVD sets DVD, cable, TV Setting alarm clock sets heat, coffee, car
Consistency for Users And user preferences
More devices UPnP Etc.
Brad Myers Carnegie Mellon 43
For More Information
Many papers on the Pebbles web site On PUC and other work
Most programs available for free downloading:
SlideShow Commander islicensed for commercial sale See handango.com
Many papers on the Pebbles web site On PUC and other work
Most programs available for free downloading:
SlideShow Commander islicensed for commercial sale See handango.com
http://www.cs.cmu.edu/~pebbleshttp://www.cs.cmu.edu/~pebbles
Brad Myers Carnegie Mellon 44
Conclusions
As more and more electronics are computerized and communicating, mobile devices can help control them
Handheld devices can improve the user interfaces of everything else Not just be another gadget to be learned
As more and more electronics are computerized and communicating, mobile devices can help control them
Handheld devices can improve the user interfaces of everything else Not just be another gadget to be learned
Brad Myers Carnegie Mellon 45
Thanks to Pebbles Students This research performed by a large number
of students since 1997:
This research performed by a large number of students since 1997:
Htet Htet AungRishi BhatnagarBen Bostwick Franklin ChenYu Shan ChuangKaren CrossCarl EvankovichIvan GonzalezMarc KhadpeDave KongChun-Kwok LeeJoonhwan Lee
Jennifer LiYuhua LiLeo LieJack LinKevin LitwackA. Chris LongColin McCabeRob MillerJeff NicholsChoon Hong PeckMathilde PignolSuporn Pongnumkul
Brandon RothrockRajesh SeenichamyPegeen ShenHerbert StielJeff StylosClaire TokarMarsha Tjandra Adrienne WarmackJacob O. WobbrockJerry YangSunny YangBrian Yeung
Brad Myers Carnegie Mellon 46
Thanks to Our Sponsors!The Pebbles research is supported by grants from:
DARPANSFMicrosoftGeneral MotorsNEC FoundationPittsburgh Digital Greenhouse
And equipment grants from: Hewlett PackardLucent TechnologiesLantronix, Inc.Lutron, Inc.MitsubishiPalm Computing
The Pebbles research is supported by grants from:DARPANSFMicrosoftGeneral MotorsNEC FoundationPittsburgh Digital Greenhouse
And equipment grants from: Hewlett PackardLucent TechnologiesLantronix, Inc.Lutron, Inc.MitsubishiPalm Computing
Symbol TechnologiesIBMSMART TechnologiesVividLogicSynergy SolutionsHandango
Symbol TechnologiesIBMSMART TechnologiesVividLogicSynergy SolutionsHandango
Mobile Devices for Control ofUbiquitous Multimedia
Mobile Devices for Control ofUbiquitous Multimedia
Brad A. [email protected]
http://www.pebbles.hcii.cmu.edu/
Brad A. [email protected]
http://www.pebbles.hcii.cmu.edu/
Human Computer Interaction Institute Human Computer Interaction Institute
School of Computer ScienceSchool of Computer Science
Carnegie Mellon UniversityCarnegie Mellon University