Context Learning Can Improve User Interaction
description
Transcript of Context Learning Can Improve User Interaction
Context Learning Can Improve User Interaction
Sushil J. Louis, Anil K. ShankarEvolutionary Computing Systems Lab (ECSL)
Department of Computer Science and EngineeringUniversity of Nevada, Reno
http://www.cs.unr.edu/[email protected]
Current UIs can be improved
• Hardware– Keyboard, mouse, clock
• Software– GUI
• Little personalization, no long term-memory
• Little use of context• Advances in speech,
vision, and text analysis have not been well integrated
Can extended context improve UI
• What sensors should we use?
• How do we use extended context to improve user interaction?– Can we personalize interaction– Personalized transportable UI
PC is a stationary robot
Simple sensors provide context• Good vision, speech
recognition, and image or speech understanding are hard AI problems
• What can we do with simple sensors?– Object recognition
versus motion detection– Speech recognition
versus speech detection– Keyboard activity– Mouse activity– Selected processes
Simple context allows richer user interaction
• If there is no one in the room should I pop up a scheduled appointment?
• If there is someone in the room should I remind Jane?
• Should I turn down my music player when the telephone rings?
• Should I pause the current song when Jane leaves the room?
…But every user has different
answers!...
SycophantSycophant uses ML techniques to learn context to action mappings
• SycophantSycophant is a calendaring application that learns to predict preferred reminder actions
• SycophantSycophant stores user interaction and context
• SycophantSycophant learns to predict reminder type
Related Work
• Reba (Kulkarni 1992) – PC is a stationary robot• Bailey and Adamczyk, 2004 – Interruptions disrupts
user’s emotional state and task performance• Hudson, Fogarty, et al, 2003 – predict interruptibility
from context. Wizard of Oz study (simulated sensors) achieved 82.4% accuracy
• SycophantSycophant learns whether or not to interrupt the user as well as how to interrupt the user
• SycophantSycophant uses real sensors
SycophantSycophant uses simple context to predict action
• Sensors for context– Keyboard, mouse– Motion:
http://motion.sourceforge.net and a cheap logitech webcam
– Speech: http://www.speech.cs.cmu.edu the Sphinx speech recognition engine. We only DETECT speech
– Five processes: java, bash, terminal, xscreensaver, mozilla
• SycophantSycophant reminder actions (Four classes)– Visual (Popup), Speech (TTS),
Neither, Both
User has to provide feedback on action
suitability
SycophantSycophant stores sensor data• For each sensor and process we store the
following data if the sensor was activated (15 sec intervals)– Any5 : any in 5 minute interval– All5 : all 5 minutes– Any1 : any in 1 minute interval– All1 : all 1 minute– Immed: in the last 15 seconds– Count : number of times sensor active in last 5
minutes
• User
((4 sensors + 5 ((4 sensors + 5 processes) X 6 processes) X 6
derived values + 1 derived values + 1 user) = 55 total user) = 55 total
featuresfeatures
SycophantSycophant uses WEKA ML tools
• Zero-R: predicts majority class
• One-R: one level decision tree testing one attribute
• J48 : Decision tree like C4.5
• Bagging: Voting over N decision trees
• LogitBoost: Numerical model
• Naïve Bayes: Bayes
Results • Performance of
decision tree inducer with different number of features
• Run J48 on all features, then choose most significant N features
• Show performance on N features with J48Not much difference in performance with fewer features
Results: Predict user action
• Performance of different ML algorithms on 25 feature data set on four class problemSmall differences in performance
Results: Two class problemClass1: Remind, Class 2: No reminder
• Significant increase in performance• From 65% to 80%
Results
• Sycophant performs at 65% on four class problem
• Sycophant performs at 80% on two class problem
• Removing motion and Removing motion and speech detectors speech detectors results in a results in a statistically significant statistically significant decrease in decrease in performanceperformance
• Sample Rules: – IF Keyboard Any5
&& speech count > 2 && no motion in last 1min && appoint time > 1220 THEN generate Speech AND Popup reminders
– IF Keyboard Any5 && speech count > 2 && keyboard Any1 THEN generate Speech only
Summary
• Sycophant uses machine learning tools to learn a mapping from user context to user actions
• Simple context provides good features• Motion and speech sensors leads to
statistically significant performance improvement
• 65% accuracy on four class problem• 80% accuracy on two class problem
Future work
• We are developing a general architectural framework for a context learning layer for all applications
• Improve performance• We need more studies with other users
and different types of users• Feature subset selection• Classifier systems
Acknowledgements
• Office of Naval Research – Contract Number N00014030104
• Evolutionary Computing System Lab (ECSL)– Chris Miles– Kai Xu– Ryan Leigh– http://ecsl.cs.unr.edu
• Anil K. ShankarAnil K. Shankar– http://www.cs.unr.edu/~anilk– Code, other papers