Post on 10-May-2015
description
USI Seminar, May 30, 2007
Pen-Based Gesture Recognition
Beat Signer
Institute for Information Systems, ETH Zurich
8092 Zurich, Switzerland
http://vub.academia.edu/BeatSigner
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 2
Outline
Motivation
Digital Pen and Paper Functionality
Gesture recognition algorithms
Evaluation of algorithms
iGesture framework
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 3
Pen-Based User Interfaces
Light Pen Graphics Tablet
PDAs and Tablet PCs Digital Pen and Paper
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 4
Palm Graffiti Gestures
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 5
Microsoft Application Gestures
scratch-out erase content
triangle insert
square action item
star action item
check check-off
curlicue cut
double-curlicue copy
circle application-specific
double-circle paste
left-semicircle undo
right-semicircle redo
caret past/insert
inverted-caret insert
chevron-left application-specific
chevron-right application-specific
arrow-up application-specific
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 6
Customised Gestures
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 7
Gesture Design Guidelines
Different gestures should not look too similar
better recognition results
Gestures should be user friendly
intuitivity and usability
Gesture set size
Reuse of gestures
same semantics for different applications
application-specific gestures
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 88
Digital Pen and Paper
Developed by Anoto
Different pen manufacturers
Logitech, Maxel, Nokia, ...
Pattern space
60 million km2
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 9
Digital Ink Processing
Parse proprietary pen formats
position, timestamp, force, tilt, …
Stroke detection
pen up / down events
time difference between succeeding points
distance between succeeding points, …
Interpolation / Filtering
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 10
Note Representation
<note>
<trace>
<point>
<x>59.664</x><y>56.364</y>
<timestamp>1103123178464</timestamp>
<force>224.0</force>
</point>
<point>
<x>59.671</x><y>56.378</y>
<timestamp>1103123178484</timestamp>
<force>227.0</force>
</point>
...
<trace>
...
</note>
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 11
Ink Markup Language (InkML), W3C
XML language for representing digital ink
Pen movement data
<trace> and <traceFormat> elements
x,y coordinates, force, angles
Device-specific information
resolution, sample rate, …
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 12
Ink Markup Language (InkML)
<ink>
<trace>10 0, 9 14, 8 28, 7 42, 6 56, 6 70, 8 84, 8 98,
8 112, 9 126, 10 140, 13 154, 14 168, 17 182, 18 188
</trace>
<trace>130 155, 144 159, 158 160, 170 154, 179 143,
179 129, 166 125, 152 128, 140 136, 131 149, 126 163
</trace>
...
<inkSource manufacturer="magicomm" model="G303">
<sampleRate uniform="true" value="70"/>
<srcProperty name="weight" value="100" units="g"/>
</inkSource>
...
</ink>
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 13
Gesture Recognition Algorithms
Offline recognition algorithms
static image
Online recognition algorithms
spatio-temporal representation
Recognition methods
statistical classification, neural networks, …
Supported gesture types
single-stroke or multi-stroke
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 14
SiGeR Algorithm
Simple Gesture Recogniser (Scott Swigart)
Gestures described by regular expressions
N, NE, E, SE, S, SW, W and NW
E
W
NS
(E)+(N)+(W)+(S)+
(NE|E|SE)+(NW|N|NE)+
(SW|W|NW)+(SE|S|SW)+
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 15
SiGeR Algorithm ...
Use additional statistical information
Description = Directions [";" Constraints];
Directions = Direction ["," Directions];
Direction = "N"|"NE"|"E"|"SE"|"S"|"SW"|"W"|"NW";
Constraints = Constraint ["AND" Constraints];
Constraint = Operand Operator Operand;
Operator = "EQ"|"NEQ"|"GT"|"GTE"|"LT"|"LTE";
Operand = "N"|"NE"|"E"|"SE"|"S"|"SW"|"W"|"NW"|
"DIAGONAL"|"STRAIGHT"|"PROXIMITY"|"STOPPOINTS";
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 16
SiGeR Algorithm ...
Examples
E,N,W,S;STRAIGHT GT 0.8
AND PROXIMITY LT 0.2
E
W
NS
W,SW,SE,NE,N,S;
PROXIMITY GT 0.7
AND STRAIGHT GT 0.3
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 17
Rubine Algorithm, 1991
Statistical classification algorithm for single
stroke gestures (training / classification)
A gesture G is represented as vector of P
sample points
iiiiP tyxsssG ,,with , ,... 10
Feature vector f extracted from G
Ffff ,...1
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 18
Rubine Features
5
016
2
01
2
015
minmax
minmax4
2
minmax
2
minmax3
2
02
2
02
022
2
02
2
02
021
)(cos
arctan
)()(
)()(
)(sin
)()(
)(cos
f
xxf
yyxxf
xx
yyf
yyxxf
yyxx
yyf
yyxx
xxf
P
PP
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 19
Rubine Features …
0113
2
222
0121
22
11
2
1
10
2
1
9
11
11
2
0
22
8
11
5
017
max Let
arctanLet
Let
sin
ttf
t
yxfttt
fff
yxxx
yxyx
yxf
yyyxxx
f
yyf
P
i
iiP
iiii
P
ii
i
P
i
i
P
i
i
iiii
iiiii
P
i
ii
iiiiii
P
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 20
Rubine Training / Classification
Training phase
Recognition / classification phase
Optimal
Classifier Fccc www ˆ0ˆˆ ,...,gesture samples
for class c
F
i
iiccc fwwv1
ˆ0ˆˆ
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 21
SiGrid Algorithm
Signature algorithm
010000
010010
011010
011000
001010
001010
000010
000000
010001
010011
011011
011011
001001
001011
000011
000001
010101
010111
0111111
011111
001101
001111
000111
000101
010100
010110
011110
011110
001100
001110
000110
000100
110100
110110
111110
111100
101100
101110
100110
100100
110101
110111
111111
111101
101101
101111
100111
100101
110001
110011
111011
111001
101001
101011
100011
100001
110000
110010
111010
111000
101000
101010
10010
100000
Example
8 x 8 Grid
001101
001110
100110
101110
000110
101100
111100
111110
110110
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 22
Evaluation Grafitti Numbers
E-Rubine Rubine SiGrid SiGeR
147
3
0
134
15
1
132
15
3
133
4
13
Correct
Error
Reject
Number of gesture classes: 10
Training: 15 examples for each gesture class (collected by 1 person)
Test Samples: 150 (collected by 3 different persons)
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 23
Evaluation
RecallPrecision
RecallPrecisonMeasureF
FNTP
TPRecall
FPTP
TPPrecision
2
negative false :FN
positive false :FP
positive true:TP
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 24
Evaluation Grafitti Numbers
E-Rubine Rubine SiGrid SiGeR
147
3
0
0.980
1.000
0.990
134
15
1
0.899
0.993
0.944
132
15
3
0.898
0.978
0.936
133
4
13
0.970
0.911
0.940
Correct
F-Measure
Recall
Error
Reject
Precision
Number of gesture classes: 10
Training: 15 examples for each gesture class (collected by 1 person)
Test Samples: 150 (collected by 3 different persons)
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 25
Evaluation Grafitti Letters (1)
E-Rubine Rubine SiGrid
334
52
4
0.865
0.988
0.923
280
107
3
0.724
0.989
0.836
273
114
3
0.705
0.989
0.824
Correct
F-Measure
Recall
Error
Reject
Precision
Number of gesture classes: 26
Training: 15 examples for each gesture class (collected by 1 person)
Test Samples: 390 (collected by 3 persons)
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 26
Evaluation Grafitti Letters (2)
E-Rubine Rubine SiGrid
342
18
3
0.950
0.991
0.970
305
48
10
0.864
0.968
0.913
297
66
0
0.818
1.000
0.900
Correct
F-Measure
Recall
Error
Reject
Precision
Number of gesture classes: 26
Training: 4x4 examples for each gesture class (collected by 4 persons)
Test Samples: 363 (collected by 4 persons)
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 27
Evaluation MS Application Gestures
E-Rubine Rubine SiGrid
196
4
0
0.980
1.000
0.990
178
19
3
0.904
0.983
0.942
145
32
23
0.819
0.863
0.840
Correct
F-Measure
Recall
Error
Reject
Precision
Number of gesture classes: 40
Training: 15 examples for each gesture class (collected by 1 person)
Test Samples: 200 (collected by 1 person)
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 28
Multi-Stroke Gestures
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 29
Evaluation Multi-Stroke Gestures
E-Rubine Rubine SiGrid
75
0
0
1.000
1.000
1.000
72
3
0
0.960
1.000
0.980
73
2
0
0.973
1.000
0.986
Correct
F-Measure
Recall
Error
Reject
Precision
Number of gesture classes: 15
Training: 15 examples for each gesture class (collected by 1 person)
Test Samples: 75 (collected by the same person)
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 30
iGesture Framework
Exercise
download iGesture
analyse the design of the
framework
investigate the existing
algorithms
design and test your own
algorithm
build a small application
based on your recogniser
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 31
iGesture Architecture Overview
Common Data Structures
Management
Console
Evaluation
Tools
Recogniser
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 32
Gesture Representation
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 33
Recognition Algorithm
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 34
Persistence
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 35
iGesture Test Bench Tab
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 36
iGesture Admin Tab
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 37
iGesture Test Data Tab
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 38
Capture of Samples
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 39
Evaluation Tools
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 40
Recogniser API
Configuration configuration =
XMLTool.importConfiguration(new File(RUBINE));
GestureSet gestureSet =
XMLTool.importGestureSet(new File(RUBINE_SET));
configuration.addGestureSet(gestureSet);
recogniser = new Recogniser(configuration);
...
ResultSet result = recogniser.recognise(note);
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 41
Conclusions
Digital pen and paper technology
Gesture recognition algorithms
SiGeR, Rubine (E-Rubine), SiGrid
Evaluation
precision, recall and F-measure
General iGesture recognition framework
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch
References
Beat Signer, Ueli Kurmann and Moira C. Norrie,
iGesture: A General Gesture Recognition
Framework, Proc. of ICDAR 2007, 9th Intl.
Conference on Document Analysis and
Recognition, Curitiba, Brazil, September 2007
42
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch
References ...
Beat Signer, Moira C. Norrie and Ueli Kurmann,
iGesture: A Java Framework for the
Development and Deployment of Stroke-Based
Online Gesture Recognition Algorithms,
Technical Report ETH Zurich, TR561,
September 2007
43
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 44
References ...
Dean Rubine, Specifying Gestures by Example,
Proc. of ACM SIGGRAPH '91, 18th Intl.
Conference on Computer Graphics and
Interactive Techniques
iGesture Open Source Framework
http://www.igesture.org