PhonePoint Pen: Using Mobile Phones to Write

in Air

Sandip Agrawal, Ionut Constandache, Shravan Gaonkar,

Romit Roy Choudhury

ACM MobiHeld 2009

Noting small pieces of information,

quickly and effortlesslycan be useful

But aren’t there several solutions ...

Buy Milk

State of the Art

Sticky notes … Difficult to organize electronicallyDifficult to organize electronically Not always handyNot always handy

Typing on phone keyboards

Tiny keyboard sizesTiny keyboard sizes Small inter-key spacingSmall inter-key spacing

Audio recording Cannot sketch diagramsCannot sketch diagrams Time consuming to browse through voice messagesTime consuming to browse through voice messages

So, need a solution that is

Easy to use

Always-with-me

Allows sketching

Online/Searchable QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Proposed Approach: PhonePoint Pen

User holds phone like a penUser holds phone like a pen Writes messages in air

Built-in accelerometer Captures hand movementCaptures hand movement Converts to image / textConverts to image / text

Display text on screen Also sent to user’s email addressAlso sent to user’s email address

Use Cases

Mashing with cameras Superimposing text on picturesSuperimposing text on pictures

Sketching Giving directions to your home Giving directions to your home Exchanging figures on the flyExchanging figures on the fly

One handed use E.g., carrying a suitcaseE.g., carrying a suitcase

Duke

University

Use Cases

Applications in healthcare Hand movement disabilitiesHand movement disabilities Speech problems after surgerySpeech problems after surgery

Military and emergency response

Quick and silent updatesQuick and silent updates

Prototype

Current prototype on Nokia N95 Equipped with 3-axes accelerometerProgrammed usign Python + MATLAB

several design challenges emerge …

Design Challenges (1)

Hands rotate while writing Accelerometers only measure linear Accelerometers only measure linear acceleration acceleration Rotation injects ambiguity Rotation injects ambiguity (Wii uses gyroscope)(Wii uses gyroscope)

Proposed Approach: • Hold phone in non-rotating gripHold phone in non-rotating grip• Determine angular orientation during the Determine angular orientation during the

pausepause

Design Challenges (2)

Background Vibration (Noise) Sensitive accelerometersSensitive accelerometers Jitter from hand vibrationsJitter from hand vibrations

Proposed Approach: • Smooth the accelerometer readings with Smooth the accelerometer readings with

moving averagemoving average• Suppress acceleration values < thresholdSuppress acceleration values < threshold

Design Challenges (3)

Computing Displacement Ambiguity when acceleration is zeroAmbiguity when acceleration is zero Can be constant velocity or stationary Can be constant velocity or stationary Worse with noiseWorse with noise

Proposed Approach: • Detect change of direction Detect change of direction • Reset velocity between strokesReset velocity between strokes Reset

Design Challenges (4)

Lifting Pen from the Paper e.g., dotting the “i” and crossing the “t”e.g., dotting the “i” and crossing the “t”

Proposed Approach: • Impulse on the Z axis during the liftImpulse on the Z axis during the lift• Off the Record – On the RecordOff the Record – On the RecordP

Z-axisspikes

Design Challenges (5)

Character transition e.g., B vs 13e.g., B vs 13

Proposed Approach: • Use delimited such as “.”Use delimited such as “.”• 1.3 for 13 1.3 for 13

Drawing a Rectangle

RawRaw Accelerometer ReadingAccelerometer Reading

Noise Smoothing using Noise Smoothing using Moving AverageMoving Average

Background Noise Background Noise SuppressionSuppression

Velocity Plot after Avoiding Velocity Plot after Avoiding Velocity DriftsVelocity Drifts

Final RectangleFinal Rectangle

Raw Accelerometer Data for M and iRaw Accelerometer Data for M and i

Character Recognition

More PhonePen words …

More PhonePen words …

20 words tested with online OCR web service.78% accuracy

20 words tested with online OCR web service.78% accuracy

Results

Video Demo

QuickTime™ and a decompressor

are needed to see this picture.

Of Course, Not a Product Yet

Lowercase character recognition Cursive handwriting more complicated

Need smaller hand movements in air

Youtube comment: looking crazy waving hands Slashdot: Nearby people can get injured

Need increased robustness 20% failure when writing in air 5% failure when writing on a surface

Of Course, Not a Product Yet

What about longer messages Sentences? Email replies?

Erasing alphabets Shake your hands like using a eraser

Ongoing work is pursuing

these directions …

Ongoing work is pursuing

these directions …

Swipe your credit card and sign in air …

The vision is:

Thanks

Visit Systems Networking Research Group (SyNRG)

@ Duke University

Google “synrg duke”

Thank You!

email: romit@ee.duke.edu

Web: http://synrg.ee.duke.edu/

Noting small pieces of information,

quickly and effortlesslycan be useful

So, what are the options ...

CallJohn @ 6658

Buy Milk

Right on3rd

traffic light

Velocity Plot after Velocity Plot after Avoiding Velocity DriftsAvoiding Velocity Drifts

Final RectangleFinal Rectangle