Developing Sketch Recognition and Interaction Techniquesfor Intelligent Surfaceless Sketching User Interfaces

Paul TaeleSketch Recognition LabTexas A&M University

College Station, TX 77839 USAptaele@cse.tamu.edu

Tracy HammondSketch Recognition LabTexas A&M University

College Station, TX 77839 USAhammond@cse.tamu.edu

ABSTRACTAs commercial motion-tracking sensors achieve greaterreliability and ubiquity, intelligent sketching user interfacescan expand beyond traditional surface environments forricher surfaceless sketching interactions. However, relevanttechniques for automatically recognizing sketches in surface-less interaction spaces are either largely constrained, dueto limited gesture input vocabularies from existing gesturerecognition techniques; or unexplored, due to being adaptedspecifically for surface environments by existing sketchrecognition techniques. This dissertation research thereforeproposes to investigate techniques for developing intelligentsurfaceless sketching user interfaces. The core researchwork will focus on investigating automated recognitiontechniques for better understanding the content of surfacelesssketches, and determining optimal interaction techniques forimproving related intuitive sketching cues in those surfacelessinteraction spaces.

Author KeywordsSketch recognition; natural user interfaces; surfacelessinteraction.

ACM Classification KeywordsH.5.2. Information Interfaces and Presentation (e.g. HCI):User Interfaces - Interaction Styles

CONTEXT AND MOTIVATIONSketching on digital devices has conventionally involvedusers interacting with pen- and touch-sensitive screens suchas on tablets or stylus-enabled monitors. However, withgreater ubiquity and reliability from emerging commercialmotion-tracking sensors, developers have greater resources toexpand beyond intelligent surface sketching user interfacesthrough surfaceless sketching. The potential of such intel-ligent surfaceless sketching user interactions can introduceusers to experience novel and intriguing interaction scenarios,such as designing three-dimensional artifacts more fluidly,

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the fullcitation on the first page. Copyrights for third-party components of this work mustbe honored. For all other uses, contact the owner/author(s). Copyright is held by theauthor/owner(s).IUI’14, February 24–27, 2014, Haifa, Israel.ACM 978-1-4503-2184-6/14/02.http://dx.doi.org/10.1145/2559184.2559185

(a) (b)

(c) (d)

Figure 1: Two types of surfaceless sketching scenarios andpossible setups: (a-b) mid-air sketching with interactionentirely in the air using a Microsoft Kinect and largetelevision, and (c-d) above-surface sketching complementingsurface sketching using a Leap Motion and tablet.

engaging in communicative sketches more expressively,and interacting in augmented reality environments moreuniquely. Existing motion-tracking sensors and interactiontechniques are already able to allow users to performnaı̈ve sketching interactions in surfaceless environments,including mid-air interaction spaces for sketching solelyin the air (Figure 1a) and above-surface interaction spacesfor optionally complementing surface sketches (Figure 1c).However, adapting intelligent user interfaces to handle richersketching interactions that are both recognized and alsoperformed intuitively present existing challenges. In theformer, relevant techniques are either constrained, due toexisting gesture recognition techniques’ limited symbolicgesture input vocabularies; or unexplored, due to existingsketch recognition techniques focused exclusively on on-surface sketches. For the latter, sketching in surfacelessenvironments introduces additional interaction characteristicsnot present in surface environments.

RELATED WORKImprovements in motion-tracking hardware have enabledresearchers to explore various interaction techniques in sur-

1

IUI 2014 • Doctoral Consortium February 24–27, 2014, Haifa, Israel

53

(a) (b)

(c) (d)

Figure 2: Interaction challenges that impact surfacelesssketch recognition: (a) surfaceless analogs, (b) imprecisesensors, (c) optimal contexts, and (d) surfaceless factors.

faceless environments. For example, work by Chan et al. [2]has investigated interaction techniques for successfullyemulating on-surface interactions on off-surface intangibledisplays, while work by Gustafson [3] has similarly exploredtouchscreen interactions such as drawing on screenlessmobile devices.

In extending these surfaceless sketching interaction tech-niques to intelligent sketching interfaces, one related areastems from existing sketch recognition works such asShortStraw [12], which handles segmentation of raw surfacestrokes; PaleoSketch [8], which recognizes these segmentsinto primitive geometric shapes, and LADDER [4], whichlater recognizes the entire surface sketch from this collectionof shapes. While these techniques are successful for surfacespaces, they remain largely untested for surfaceless spaces.

Similarly, existing surfaceless gesture recognition techniquesfrom $3 [5] and Protractor3D [6], which are three-dimensional adaptations of surface gesture recognitiontechniques $N [1] and Protractor [7], respectively, explorethe automated understanding of simplified sketch-like inter-actions, but their constrained gesture input vocabularies [9]make them less suitable for richer and more diversesurfaceless sketched input.

RESEARCH GOALS AND METHODSThe research goals and proposed methods for the dissertationresearch were categorized based on preliminary observationsand users’ feedback from our pilot studies, which naivelyadapted existing surface sketch recognition techniques ontosurfaceless environments [10, 11].

Goal #1: Determining Surfaceless AnalogsWhile intelligent user interfaces primarily rely on stylusesor fingers to perform surface sketching on touch-enabledscreens, determining intuitive analogs for surfacelesssketching is less clear and requires further consideration due

to the lack of both straightforward input modalities and anexplicit surface medium. Accomplishing this goal thereforerequires addressing the following research question: Whatinteraction cues and input modalities can we determineto be appropriate surfaceless analogs for sketching? Theproposed research plans to address this goal by exploringsurfaceless sketching on various candidate input modalitiesand also investigating intuitive sketching cues throughquantitative interaction design metrics (e.g., Fitts’ Law) andqualitative user feedback (e.g., Likert scales).

Goal #2: Addressing Imprecise SensorsImprovements in commercial motion-tracking sensors areexpanding possible surfaceless sketching scenarios, suchas mid-air sketching interactions with a large display andMicrosoft Kinect (Figure 1b) and above-surface sketchinginteractions with a tablet and Leap Motion (Figure 1d).However, motion-tracking sensors are non-trivially lessprecise than touchscreens for detecting raw points insketches, while existing sketch recognition techniques relyentirely on these more precise surface sketching assumptions.Accomplishing this goal therefore requires addressing thefollowing research question: How do we address imprecisehardware sensor information for robustly recognizingsketches made in surfaceless interaction spaces? Theproposed research plans to address this goal by exploringways to adapt existing surface sketch recognition techniquesfor more complex surfaceless environments.

Goal #3: Discovering Optimal ContextsSketching beyond surface environments affords an additionalspatial dimension and a relatively larger interaction space,and therefore motivates potential creative scenarios thatcan expand the interactions of existing surface sketchingenvironments and develop novel ones for surfacelessenvironments. However, there may be some surface sketchingscenarios that do not translate as well for surfacelesssketching, such as domains that require detailed precision(e.g., technical schematics) or numerous annotations (e.g.,note taking). Accomplishing this goal therefore requiresaddressing the following research question: What domaincontexts can be discovered that are appropriate forsketching in surfaceless environments? The proposedresearch plans to adapt existing sketching domains insurfaceless environments and evaluate their effectivenessabsent of surface interaction assumptions.

Goal #4: Accommodating Surfaceless FactorsIn addition to determining appropriate interaction cues andinput modalities for surfaceless sketching analogs, thereare also other factors that may affect surfaceless sketchrecognition and interactions. These contributing factorsinclude encountering limb fatigue from prolonged sketching(i.e., Gorilla Arm Syndrome) and determining whethersketching is actively occurring or not in mid-air (i.e., MidasTouch Problem). Accomplishing this goal therefore requiresaddressing the following research question: How do weeffectively accommodate contributing factors that affectsketching in surfaceless environments? The proposed

2

IUI 2014 • Doctoral Consortium February 24–27, 2014, Haifa, Israel

54

Figure 3: Preliminary recognition results for primitive geo-metric shapes categorized as either polylines or curvatures.

research plans to first observe how these contributing factorsaffect recognition and interaction techniques for surfacelesssketching, and then adjust these techniques to properlyaccommodate these factors.

DISSERTATION STATUSWe initially approached the dissertation research’s goalsby measuring the performance of successful low-levelsurface sketch recognition techniques onto a surfacelessinteraction space, in order to determine whether thoseexisting techniques could be improved upon. Our pilot studytherefore involved comparing the recognition accuracies ofprimitive shapes on standard baseline features Speed andCurvature, state-of-the-art baseline segmenter ShortStraw,and our prototype segmenters BasicFit and BasicFit+with thresholds naı̈vely adapted for surfaceless scenarios.We recruited four users with prior interaction experienceinteracting with motion-tracking sensors to sketch variationsof polylines and curvatures with a Microsoft Kinect on alarge television display. The interactions were designed suchthat users sketched with their dominant arm and activatedand edited sketches with their non-dominant arm, since thefocus of the pilot study was solely on obtaining surfacelesssketches. Our preliminary results (Figure 3) used ten-foldcross-validation and all-or-nothing recognition, and showedthat our proposed segmenters performed better in recognizingpolylines and curvatures compared to existing surface sketchfeatures and segmenters. These results indicated hopefulpotential in further adapting surface sketch techniques forsurfaceless interaction spaces.

The results described have already been published as seedresearch abstracts for peer-reviewed venues in human-computer interaction [10] and artificial intelligence [11].I have also written an outline and rough draft of mydissertation’s motivation and prior work sections, and willcomplete my preliminary examination by December 2013.

NEXT STEPSThe subsequent steps will specifically address the previouscore dissertation research goals listed. For determiningcorresponding surfaceless analogs, the next step aims to in-vestigate and evaluate possible sketching cues for surfacelesssketching from several candidate input modalities, such as anempty hand, a wireless mouse, a tablet stylus, and a wandremote. For addressing imprecise hardware sensors, the nextstep aims to continue adapting existing sketch recognition

techniques onto surfaceless assumptions, especially for low-level geometric shape classification and high-level wholesketch recognition. For discovering optimal domain contexts,the next step aims to implement intelligent surfacelesssketching user interfaces for domains with existing sketchingcomponents (e.g., drawing designs, sketching diagrams), andthen evaluating those interfaces’ usability for those domainswith surfaceless sketching assumptions. For accommodatingcontributing surfaceless factors, the next step aims to closelyobserve users’ surfaceless sketching for any factors thatnegatively impact recognition and interaction performance,and then experimenting with various approaches to alleviatethese factors.

REFERENCES1. Anthony, L., and Wobbrock, J. O. $n-protractor: a fast

and accurate multistroke recognizer. In Proc. GI 2012,Canadian Information Processing Society (2012),117–120.

2. Chan, L.-W., Kao, H.-S., Chen, M. Y., Lee, M.-S., Hsu,J., and Hung, Y.-P. Touching the void: direct-touchinteraction for intangible displays. In Proc. CHI 2010,ACM (2010), 2625–2634.

3. Gustafson, S. Imaginary interfaces: touchscreen-likeinteraction without the screen. In Proc. CHI 2012, ACM(2012), 927–930.

4. Hammond, T., and Davis, R. Ladder, a sketchinglanguage for user interface developers. Comput. Graph.29, 4 (Aug. 2005), 518–532.

5. Kratz, S., and Rohs, M. A $3 gesture recognizer: simplegesture recognition for devices equipped with 3dacceleration sensors. In Proc. IUI 2010, ACM (2010),341–344.

6. Kratz, S., and Rohs, M. Protractor3d: a closed-formsolution to rotation-invariant 3d gestures. In Proc. IUI2011, ACM (2011), 371–374.

7. Li, Y. Protractor: a fast and accurate gesture recognizer.In Proc. CHI 2010, ACM (2010), 2169–2172.

8. Paulson, B., and Hammond, T. Paleosketch: accurateprimitive sketch recognition and beautification. In Proc.IUI 2008, ACM (2008), 1–10.

9. Steins, C., Gustafson, S., Holz, C., and Baudisch, P.Imaginary devices: gesture-based interaction mimickingtraditional input devices. In Proc. MobileHCI 2013,ACM (2013), 123–126.

10. Taele, P., and Hammond, T. Initial approaches forextending sketch recognition to beyond-surfaceenvironments. In Proc. CHI 2012, ACM (2012),2039–2044.

11. Taele, P., and Hammond, T. Adapting surface sketchrecognition techniques for surfaceless sketches. In Proc.IJCAI 2013, AAAI (2013), 3243–3244.

12. Wolin, A., Eoff, B., and Hammond, T. Shortstraw: asimple and effective corner finder for polylines. In Proc.SBIM 2008, Eurographics Association (2008), 33–40.

3

IUI 2014 • Doctoral Consortium February 24–27, 2014, Haifa, Israel

55