deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research...
Transcript of deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research...
![Page 1: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/1.jpg)
deWristified: Handwriting Inference Using Wrist-Based
Motion Sensors Revisited
Raveen [email protected]
Anindya [email protected]
Murtuza [email protected]
University of Texas at San Antonio
![Page 2: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/2.jpg)
Wrist Wearables
• Extends the functionality of traditional wristwatches beyond timekeeping.
• Captures rich contextual information about the wearer.• Enables several novel context-based applications.
2019-05-16 SPriTELab @ UTSA 2
![Page 3: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/3.jpg)
Motion Sensors
• Two main types of motion or inertial sensors:• Accelerometer: records device acceleration.• Gyroscope: records device angular rotation.
• Accessing motion sensors on wearable devices:
• All applications have access to motion sensors by default (also referred to as zero-permission sensors) on most wearable OSs.
• Applications’ access to motion sensors cannot be regulated on most wearable OSs –we can’t turn them off!
• Can an adversary take advantage of motion sensor data from a wrist-wearable device to infer private information inputted by the user’s device-wearing hand?
2019-05-16 SPriTELab @ UTSA 3
![Page 4: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/4.jpg)
Inferring Private User Inputs (Using Wrist Wearables)
2019-05-16 4SPriTELab @ UTSA
![Page 5: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/5.jpg)
State-of-the-Art in Handwriting Recognition (Using Wrist Wearables)
2019-05-16 5SPriTELab @ UTSA
Airwriting (Amma et al.) Whiteboard writing (Arduser et al.)
Finger writing (Xu et al.) Pen(cil) writing (Xia et al.)
![Page 6: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/6.jpg)
Adversary Model
• Adversary has knowledge of the type of handwriting.
• Adversary is able to record data from the target smartwatch’s accelerometer and gyroscope sensors.• Could employ a Trojan app for this!
• Adversary’s Goal: To infer handwritten information using target user’s smartwatch sensors.
2019-05-16 SPriTELab @ UTSA 6
![Page 7: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/7.jpg)
Limitations of Earlier Handwriting Recognition Studies (Using Wrist Wearables)
• Airwriting (Amma et al.)• Custom-designed hand glove with very
high precision sensors.• Our adversary relies on target user’s
smartwatch or fitness band.
• Only uppercase words.
• Whiteboard writing (Arduser et al.)• Not generalized (training and testing
data not from different participants).
• Only uppercase alphabets.
• No handwriting activity detection.
• Finger writing (Xu et al.)• Use of Shimmer, a specialized sensing
device intended for lab studies.
• Not generalized (training and testing data not from different participants).
• Pen(cil) writing (Xia et al.)• Only lowercase alphabets.
• Controlled data collection.
• No handwriting activity detection.
2019-05-16 SPriTELab @ UTSA 7
![Page 8: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/8.jpg)
Our Research
• How practical is handwriting inference when• Using consumer-grade wrist
wearables,
• Using generalized training and testing,
• Writing in a uncontrolled and unconstrained manner, and
• Both upper and lowercase alphabets are modeled ?
2019-05-16 SPriTELab @ UTSA 8
New Uncontrolled and Unconstrained
Writing Data
Existing Models
![Page 9: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/9.jpg)
Handwriting Inference Framework
2019-05-16 9SPriTELab @ UTSA
![Page 10: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/10.jpg)
Experimental Setup
• 28 participants for the four writing scenarios.• 18 to 30 years of age• 13 male, 15 female
• Two different wrist-wearables.• Sony Smartwatch 3, LG Watch Urbane
• Accelerometer and gyroscope recorded at 200Hz.
• Participants provided with appropriate writing apparatus.
2019-05-16 SPriTELab @ UTSA 10
![Page 11: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/11.jpg)
Writing Tasks (In-Lab)
• Alphabets. • Individual alphabets one at a time.
• Covered all 26 English alphabets in random order.
• Each alphabet was written 10 times.
• Both upper and lower cases.
• Words.• 4-8 alphabet words, from a vocabulary (Goldhahn et al. 2012).
• Each participant wrote 20 words, in both upper and lower cases.
• Sentence.• "the five boxing wizards jump quickly" in both upper and lower cases.
2019-05-16 SPriTELab @ UTSA 11
![Page 12: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/12.jpg)
Writing Activity Recognition (Out of Lab)
• 2 participants.
• Wore a smartwatch for an entire day.
• Performed the four writing scenarios at random times.
• Adversary’s Goal: To infer handwriting activity first, and then classify the handwritten text.
2019-05-16 SPriTELab @ UTSA 12
![Page 13: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/13.jpg)
Replicated Inference Frameworks
• Airwriting • Hidden Markov Model (HMM)
• Whiteboard writing • Dynamic Time Warping (DTW)
• Finger writing • Naive Bayes, Logistic Regression and Decision Tree classifiers
• Pen(cil) writing• Random Forest classifier
2019-05-16 SPriTELab @ UTSA 13
![Page 14: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/14.jpg)
Personalized Inference Accuracy
2019-05-16 SPriTELab @ UTSA 14
Writing Activity Detection:56% recall and 57% precision for air and finger writing39% recall and 47% precision for pencil writing23% recall and 34% precision for whiteboard writing
![Page 15: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/15.jpg)
Personalized Inference Accuracy(Whiteboard Writing)
2019-05-16 SPriTELab @ UTSA 15
Lowercase Uppercase
![Page 16: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/16.jpg)
Generalized Inference Accuracy
2019-05-16 SPriTELab @ UTSA 16
Writing Activity Detection:35-40% recall for airwriting, whiteboard writing and pencil writingOnly 8% recall for finger writing
![Page 17: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/17.jpg)
Factors Affecting Inference Accuracy
• Number of Strokes.
2019-05-16 SPriTELab @ UTSA 17
![Page 18: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/18.jpg)
Factors Affecting Inference Accuracy
2019-05-16 SPriTELab @ UTSA 18
Number of strokes for the same letter for different participants (lowercase).
Number of strokes for the same letter for different participants (uppercase).
![Page 19: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/19.jpg)
Factors Affecting Inference Accuracy
2019-05-16 SPriTELab @ UTSA 19
Lowercase Uppercase
Variance in number of strokes per alphabet per participant, averaged for all participants
![Page 20: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/20.jpg)
Factors Affecting Inference Accuracy
• Number of Strokes.
• Order of Strokes.
• Direction of Strokes.
2019-05-16 SPriTELab @ UTSA 20
![Page 21: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/21.jpg)
Factors Affecting Inference Accuracy
2019-05-16 SPriTELab @ UTSA 21
![Page 22: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/22.jpg)
Factors Affecting Inference Accuracy
• Number of Strokes.
• Order of Strokes.
• Direction of Strokes.
• Uppercase vs Lowercase.
• Specialized Devices.
2019-05-16 SPriTELab @ UTSA 22
Airwriting (Amma et al.)
![Page 23: deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,](https://reader034.fdocuments.us/reader034/viewer/2022050205/5f58e253a8fad72d6f564a8f/html5/thumbnails/23.jpg)
Conclusion
• We investigated how wrist-wearable based handwriting inference attacks perform in realistic day-to-day writing situations.
• Such inference attacks are unlikely to pose a substantial threat to users of current consume-grade smartwatches and fitness bands.• Primarily due to highly varying nature of handwriting.
• Replicable artifacts: https://sprite.utsa.edu/art/dewristified
2019-05-16 SPriTELab @ UTSA 23