1 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Trainable Sketch Recognizer for...
-
Upload
andrew-baker -
Category
Documents
-
view
215 -
download
1
Transcript of 1 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Trainable Sketch Recognizer for...
1 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
Trainable Sketch Recognizer forGraphical User Interface Design
Adrien Coyette1, Sascha Schimke2, Jean Vanderdonckt1 & Claus Vielhauer2
1Université catholique de Louvain (UCL)School of Management (IAG)Information Systems Unit (ISYS)Belgian Lab. of Computer-Human Interaction (BCHI)http://www.isys.ucl.ac.be/bchi
2Otto-von-Guericke Universität MagdeburgFakultät für InformatikInstitut für Technische und Betriebliche Informationssysteme (ITI) http://wwwiti.cs.uni-magdeburg.de/
2 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
Considerable importance of user interface in application development (50% to 70 % of the application code [Myers and Al, 2000] )
Apparition of many high fidelity graphical user interfaces editors on the market
Motivations
3 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
But designing the right User Interface (UI) the first time is very unlikely to occur
instead, UI design is eminently open iterativeiterative and incompleteincomplete
Motivations
4 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
Paper ( or whiteboard) used most of the time:familiar and unconstrained approach
fast to learn and quick to produce
focus on basic structural issues instead of unimportant details
it is very appropriate to convey ongoing, unfinished designs, and it encourages creativity,
can be performed collaboratively between designers and end-users
Motivations
5 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
Motivations: empirical findings
UI Drawing or sketching increases participatory design and subjective satisfaction with respect to UI editors
[Landay & Myers, 1995, Plimmer & Apperley, 2004, Buxton, 2005]
Equal usability problems found for a sketched UI than with an edited one[Virzi et al., 1996]
Explanatory power of a sketched UI does not decrease with respect to edited one; tool support is preferred over paper prototyping
[Walker et al., 2002 ]
A sketched UI delivers same quantity and quality of outputs than an edited one[Sefelin et al., 2006]
UI sketching fosters alternative designs and communication [Tohidi et al., 2006]
Supports throw-away prototyping due to low cost[Schneider & Petrie, 2006, Bailey & Konstan, 2003]
6 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
The idea of developing a computer-based tool for sketching UIs naturally emerged from these observations
Such tools would extend the advantages provided by sketching techniques (easily creating, deleting, updating or moving UI elements).
Apparition of hybridhybrid approachesapproaches, combining the best of the hand-sketching and computer assisted user interface design
Related Works
7 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
State of the Art:Eliciting requirements
17 different software and approaches surveyedSources: scientific literature and commercial
softwareComparative analysis relying on consistent
analysis gridTool identification (7)Install and first use (4)Tool functions (8)Shape recognition (4)Shape interpretation (7)UI editors (9)
Applied to the same case study
(eCommerce app.)
8 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
DENIM(James A. Landay, James Lin, Mark W. Newman, Jason I. Hong)
Related works
9 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
Advantages:• Support for scenario-based design• Several levels of granularity• Good documentation• Toolbox of generic widgets• Mature product, based on experimental testing • Zooming facility from local design (e.g. a web page) to a global
design (e.g., a portion of a web site or n entire web site)• Storyboarding facilities based on patterns
Shortcomings:Shortcomings:• No shape recognition and interpretation, thus loosing the effort• No code generation• No preview mode • Only dedicated to web sites
Related works
10 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
JavaSketchIt (Joaquim A. Jorge, Manuel João Fonseca, Anabela Caetano, Néri Goulart )
Motivations / Related works
11 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
Advantages:• Performance (speed and accuracy)• Multi-stroke gestures• Recognizes rotated shapes • Computationally light • Open source • Requires standard and freely available libraries
Shortcomings:• Mono-window• No scenario editor• Only generates java • Limited widget set• Shape interpretation can only take as input a construct
made of maximum two vectorial shapes
Motivations / Related works
12 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
Avoidance of Effort loss
Language neutrality
Ease of use (naturalness)
Flexible processing
Robust scenario editor
Large conceptual coverage
…
Motivations / Related works
13 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
SketchiXML
14 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
SketchiXML
15 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
SketchiXML: Shape Recognition
Recognition engine: Cali libraryRecognition engine: Cali library
A fast, simple and compactA fast, simple and compact approach to identify Scribbles (multi-stroke geometric shapes)
Combines temporal adjacency, Fuzzy Logic and geometric temporal adjacency, Fuzzy Logic and geometric featuresfeatures to classify scribbles.
Recognizes elementary geometric shapes, such as Triangles, Rectangles, Diamonds, Circles… and some gesture commands, such as, Delete, Cross, WavyLine, Move, Copy and Tap.
Shapes are recognized independently of changes in rotation, recognized independently of changes in rotation, size or number of individual strokessize or number of individual strokes
The recognizer has a recognition rate of 94%94% and a fast response
16 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
SketchiXML: Shape Recognition
All widgets were based on a set of shape All widgets were based on a set of shape primitivesprimitives
Not extendableRequire a delay between each strokesLimits the number of representation
17 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
SketchiXML: Shape Recognition
Trainable Gesture RecognizerTrainable Gesture Recognizer Handdrawn input (sketch) is
superimposed with a grid Each sampling point is mapped to
its closest grid node For adjacent nodes, a stroke
direction (out of 8 different directions) is taken
A string is generated from direction coding of input
Complex task of comparison of drawing inputs is reduced to simple calculation of string edit distances
18 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
SketchiXML: Shape Recognition
Distance is shortest sequence of edit commands that transform s to t Simplest set of operations:
• Copy character from s over to t • Delete a character in s (cost 1) • Insert a character in t (cost 1) • Substitute one character for another (cost 1)
Levenstein Levenstein distance distance
19 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
SketchiXML: Shape Recognition
20 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
SketchiXML: Grammar
<widget type="ListBox"> <representation id="0">
<constraint id="0" shape1="Triangle_3" shape2="Rectangle_0" condition="isInsideInUpperRightCorner" /> <constraint id="1" shape1="Triangle_4" shape2="Rectangle_0" condition="isInsideInLowerRightCorner" />
<shape id="Rectangle_0" type="Rectangle" /> <shape id="Triangle_3" type="Triangle" /> <shape id="Triangle_4" type="Triangle" /> </representation><representation id="1"><constraint id="0" shape1="Triangle_0" shape2="Rectangle_2" condition="isInsideInTop"/>
<constraint id="1" shape1="Triangle_1" shape2="Rectangle_2" condition="isInsideInBottom" /> <constraint id="2" shape1="Rectangle_2" shape2="Rectangle_3" condition="isInsideOnTheRight" />
<shape id="Triangle_0" type="Triangle" /> <shape id="Triangle_1" type="Triangle" /> <shape id="Rectangle_2" type="Rectangle" /> <shape id="Rectangle_3" type="Rectangle" /> </representation> <representation id="2"><constraint id="0" shape1="Line_0" shape2="ListBox_1" condition="isInside" /> <constraint id="1" shape1="Line_0" shape2="-" condition="isHorizontal" /> <shape id="Line_0" type="Line" /> <shape id="ListBox_1" type="ListBox" /> </representation></widget>
21 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
Conclusion / Future Works
Conclusionan innovative contribution to the domain of sketch based
design tools: a trainable multi-strokes gesture recognizer Improved the performance of SketchiXML
Future works
evaluating other feature types to describe the gesture shape, e.g. triangular grids
Improving the performances of the algorithm, when running with a large set of representations
Conduct a systematic comparative test with other shape recognition algorithms
22 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007
Thank you very much for your attention
For more information and downloading,http://www.isys.ucl.ac.be/bchi
http://www.usixml.orgUser Interface eXtensible Markup Language
http://www.similar.ccEuropean network on Multimodal UIs
http://www.usixml.org/index.php?view=page&idpage=29SketchiXML home page