quill: A UI Design Case StudyA. Chris LongPostdoctoral Research FellowHCI Institute

AdministriviaTurn in Homework #1Homework #2 assignment on course website

OutlineGesturesquill Designquill Architecture & ImplementationObject-Oriented DesignLong-running Tasks and InterfacesRecognition-based UIsHow Gesture Recognition WorksMore quill Implementation

GesturesMark that invokes a command

GesturesMark that invokes a command

delete me

GesturesMark that invokes a command

BenefitsFastSave screen spaceAvoid aerobic interfaceSpecify operation and operand(s) simultaneouslydelete me

Problems With GesturesFor usersComputer often misrecognizes gesturesDifficult to learn and remember gesturesFor designersGestures are hard to design

quills Place in the WorldWho: Designer of pen-based UIWhen: During design phaseWhat: Designing gesturesWhy: To design better gesturesDesignPrototypeTest

Another View of quill

Designing quillDesign low-fi prototypePilot test low-fiRedesignImplement Java version

Low-fi Prototype

Current Version

Tree view

Expanded view

Suggestion for improvement

Recognition Problem

Gestures, Packages, and Groups, Oh My!cutEditMyApplicationViewcopyGestureGesture categoryGesture groupGesture package

quill ArchitectureJDK CoreGesture ObjectsGestureGesture CategoryRecognizerGesture AnalysesWidgets

quill Architecture

WidgetsGesture AnalysesGesture & Recognition ObjectsSwing Core JDK

Implementation OverviewJava/Swing93 classes, 26,000 lines

Object Oriented DesignGesture ObjectsGestureGestureCategoryGestureGroupGesturePackageGesture Display WidgetsGestureDisplayGestureInteractorGestureCategoryDisplayGestureGroupDisplayGesturePackageDisplayGestureCategoryThumbnailDisplay

OO DesignGestureObjectDefaultGestureObjectGestureContainerAbstractGestureContainerGestureCategoryGestureGroupGesturePackage

Analyzing GesturesProblem: Analysis may take a long timeSolutions?

Analyzing GesturesProblem: Analysis may take a long timeStrategies for analyzing gesturesRun analysis in foreground, disable all actionsAnalyze in backgroundDisable all conflicting actions Allow anything, cancel analysesAllow anything

MultithreadingBenefits?

Drawbacks?

MultithreadingBenefitsComputation in backgroundCan do very complex tasksUser can perform any action, any timeDrawbacksMust synchronize data across threadsDeadlockRace conditions

Multiple Threads in JavaThread classStart separate thread that runs any methodsynchronized keywordPrevents a method from being called or object from being accessed by more than one thread at onceSwingWorker classEasy to run background thread that produces a result

Recognition-Based UIsExamplesPen gestureHandwritingSpeechAdvantages?Disadvantages?

Recognition-Based UIsAdvantagesNatural inputInformalMultiple interpretationsDisadvantagesNo clear separation of data and commandsAmbiguous commandsIndividual differencesMultiple interpretations

Dealing with RecognitionConfirm dangerous operationsMultiple modes may help disambiguateSpeech + pen, for exampleProvide undo/redo

How Does Recognition Work?TrainingMeasure geometric features of gesturesCompute average feature values for each type of gestureCompute how important each feature isRecognition (a.k.a. Classification)Measure geometric features of unknown gestureCompare with features of known gesturesPick closest known gesture

Rubines Feature SetInitial angleLength of bounding box diagonalAngle of bounding box diagonalDistance between first and last pointsAngle between first and last pointsLengthTotal angle traversed

Rubines Feature Set (cont.)Sum of absolute value of angle at each pointSum of squared angle at each point (sharpness)Square of maximum speedDuration

Weakness of Feature-based RecognitionFeature set has to be right

Feature-based TrainingMean feature vector

Per-class covariance matrix

Training (cont.)Average covariance matrix

Weights estimation

Feature-based ClassificationFor all gesture classes c, compute

Return class with max vc

quill Implementation Issues (contd)Suggestions (contd)When do they disappear?User says to ignoreNo longer applyWhen are two notices the same?At time A, gestures X and Y are too similarAt time B, something happens and they arent similarAt time C, they are similar againDisplay notice again?

Implementation Issues (contd)AnnotationsSpecial purpose vs. genericHow to compose?Gesture displayDesktop windowsTiledStandard MDIDisplay factory

Implementation issues (contd)Generic propertiesvoid GestureObject.setProperty(String name, Object value)Object GestureObject.getProperty(String name)What happens to properties like isMisrecognized when copy & paste?Persistent vs. transient

Implementation issues (contd)Generic propertiesvoid GestureObject.setProperty(String name, Object value)Object GestureObject.getProperty(String name)What happens to properties like isMisrecognized when copy & paste?Persistent vs. transient

Implementation Issues (contd)Menus: enable/disable items, and dispatching commandsContext-dependent menu itemsContext-dependent gesture drawingSolution: Chain of command pattern that follows selection

Questions?Other topicsGrad schoolBerkeley (city or university)Speech UIsMore on pen-based UIs

The EndDesign Patterns: Elements of Reusable Object-Oriented Softwareby Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides

http://www.cs.cmu.edu/~chrisl/http://guir.berkeley.edu/projects/quill/

Extras

CriticismsRecognition is due to poor recognizers, which will eventually be perfectNot for a long time, if everLearning and memory predictions still usefulRecognition will never be good enoughAlready good enough for many usersIs improving over timeTyping is faster and more accurate than writingPens wont replace keyboards, but are better in many situations

Criticisms (cont.)Gestures will always be too hard to learn and remember, and will never be useful enoughGestures very often used on paper in many domainsDesign tool doesnt guarantee good gesturesWill provide useful adviceWill shorten design cycleThere are only a small number of common gesturesApplication/domain-specific gesturesPersonal shorthand

Criticisms (cont.)Marking menus are better than free-form gesturesCant specify additional information with marking-menusShapes are arbitraryAll straight lines and 45/90 angles

Criticisms (cont.)Whats wrong with traditional GUI interaction techniques?Pen mousePenFiner controlHas other dimensions (e.g., pressure, tilt)MouseMore buttons (sometimes)

Pen vs. MouseKato, et alPen significantly faster & more accurate than mouse for precise draggingPen significantly faster for pointingPen has directional dependencies; mouse doesntMackenzie, et alPen slightly faster for pointingMouse slightly faster for dragging

talk about what quill is and what it doesPoint out delete as example of specifying operation & operand simultaneously. Didnt have to select delete me first.Point out delete as example of specifying operation & operand simultaneously. Didnt have to select delete me first.Suggestions: when analyzing in background, what kind of actions do you allow?Annotations: how to show different states of gestures in different places?Desktop: tiling is difficultDisplay factory tied data type with kind of displaySuggestions: when analyzing in background, what kind of actions do you allow?Annotations: how to show different states of gestures in different places?Desktop: tiling is difficultDisplay factory tied data type with kind of display