Post on 24-Dec-2015
Using Mobile Phones to Write in Air
Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Kevin Cave, Frank DeRuyter,
Romit Roy Choudhury
More Specifically …
User holds phone like a pen Writes short messages in air
Phone’s built-in accelerometer Records acceleration due to hand motion Converts signals image text
Displays text on screen And sends a copy to user’s email address
Could be a Cool Toy
But will anyone care? Caption photos on-the-fly
Sketch maps, directions … even quick diagrams
One handed use … while carrying baby/suitcase or while driving
Duke Univ.
Let’s Build It
Prototype on Nokia N95 Equipped with 3-axes accelerometer Programmed usign Python + MATLAB
Did not expect this to be hard But we had heavily under-estimated …
Design Challenges (1)
Background Vibration (Noise) Sensitive accelerometers Jitter from hand vibrations
Approach: • Smooth signal by filtering high freq.
components• Suppress residual values < threshold
Design Challenges (2)
Computing Displacement Ambiguity when acceleration is zero Can be constant velocity or stationary Worse with noise
Approach: • Detect change of direction • Reset velocity between strokes Reset
Design Challenges (3)
Lifting Pen from the Paper e.g., dotting the “i” and crossing the “t”
Approach: • Impulse on the Z axis during the lift• Off the Record – On the RecordP
Z-axisspikes
Design Challenges (4)
Character transition e.g., B vs 13
Approach: • Use delimiters such as “.” (1.3 for 13)• Soft leftward motion between alphabets
Design Challenges (5)
Hands rotate while writing Accelerometers only measure linear acceleration Rotation injects ambiguity (Wii uses gyroscope)
Approach: • Hold phone in non-rotating grip• Determine angular orientation during the
pause
Noise Smoothing Residual Noise Suppression
Velocity Plot after Avoiding Velocity Drifts
Final Rectangle
Character Recognition
While geometric version can be displayed as image, text format necessary for searching
English text expressed in strokes:
Incorporating Tolerance to Errors
Standard “N” = I + \ + I
N = I + + I
However, imperfect human gestures induce
distortions:
N
s0
I
\
s2
I
I
DP
B R
\
Incorporating Tolerance to Errors
Standard “N” = I + \ + I
N = I + + I
However, imperfect human gestures induce
distortions:
N
s0
I
\
s2
I
I
DP
B R
\ I
Incorporating Tolerance to Errors
Standard “N” = I + \ + I
N
s0
I
\
s2
I
I
DP
B R
\ I
N = I + + I
However, imperfect human gestures induce
distortions:
Dense Grammar Tree
Incorporating Tolerance to Errors
Standard “N” = I + \ + I
N = I + + I
N = I + \ + /
N = / + \ + /
N = / + I + /…
However, imperfect human gestures induce
distortions:
Spelling Correction
English spelling correction useful Improvement possible over edit distance
MQM MOM, MUM, MAM (all edit distance = 1)
P3 mostly confuses O as Q
NIET NET, MET (edit distance = 1, 2)P3 mostly confuses M as NI
On the Shoulders of Related Work
Stylus based sketch recognition SketchREAD, Electronic Cocktail Napkin, Unistrokes
Gesture recognition uWave, SmartQuil,
Vision based gesture recognition TinyMotion, Microsoft Write in Air, VisionWand
Others … Wiimote, Air-Mouse, SmartPen, NiiMe and others
10 test users (students) 6 trained (practiced each letter once, < 5min) 4 novices Each user wrote 26 test characters in air
Results
Metrics Human readability Character recognition Word recognition (with spelling correction) Writing speed
Of Course, Not a Product Yet
Lowercase character recognition Cursive handwriting more complicated
Need smaller hand movements in air
Youtube comment: looking crazy waving hands Slashdot: Nearby people can get injured
Quicker writing 3.02 seconds/alphabet is way too slow
Of Course, Not a Product Yet
What about longer messages Sentences? Email replies?
Erasing alphabets Shake your hands like using a eraser
Ongoing work is pursuing
these directions …
Evaluation
10 test users (students) 6 trained (practiced each letter once, <5min) 4 novices Each user wrote 26 test characters
5 patients from Duke Hospital To analyze applicability into assistive
technology
Proposed Approach: PhonePoint Pen
User holds phone like a pen Writes messages in air
Built-in accelerometer Captures hand movement Converts to image / text
Display text on screen Also sent to user’s email address
Thank You!
email: romit@ee.duke.edu
Web: http://synrg.ee.duke.edu/
Noting small pieces of information,
quickly and effortlesslycan be useful
So, what are the options ...
CallJohn @ 6658
Buy Milk
Right on3rd
traffic light
Noting small pieces of information,
quickly and effortlesslycan be useful
But aren’t there several solutions ...
Buy Milk
State of the Art
Sticky notes … Difficult to organize electronically Not always handy
Typing on phone keyboards
Tiny keyboard sizes Small inter-key spacing
Audio recording Cannot sketch diagrams Time consuming to browse through voice messages
Incorporating Tolerance to Errors
Standard “N” = I + \ + I
N = I + + I
N = I + \ + /
N = / + \ + /
N = / + I + /…
However, human gestures induce
distortions:
N
Incorporating Tolerance to Errors
Standard “N” = I + \ + I
N = I + + I
N = I + \ + /
N = / + \ + /
N = / + I + /…
However, human gestures induce
distortions:
N
s0
I
\
s2
I
I
DP
B R
\
Incorporating Tolerance to Errors
Standard “N” = I + \ + I
N = I + + I
N = I + \ + /
N = / + \ + /
N = / + I + /…
However, human gestures induce
distortions:
N
s0
I
\
s2
I
I
DP
B R
\
Incorporating Tolerance to Errors
Standard “N” = I + \ + I
N = I + + I
N = I + \ + /
N = / + \ + /
N = / + I + /…
However, human gestures induce
distortions:
N
s0
I
\
s2
I
I
DP
B R
\ I
Incorporating Tolerance to Errors
Standard “N” = I + \ + I
N = I + + I
N = I + \ + /
N = / + \ + /
N = / + I + /…
However, human gestures induce
distortions:
N
s0
I
\
s2
I
I
DP
B R
\ I