Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9...
Transcript of Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9...
![Page 1: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/1.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Learning On-Air Hand Gestures From Wi-FiSignals on Smartphones
Deep Learning SeminarDr. Ramviyas Parasuraman
Assistant Professor, UGA Computer ScienceWork: Master’s thesis (2017) of Mohamed Haseeb at KTH Sweden
Slides courtesy: Mohamed from his thesis presentation.
November 8, 2019
1 / 77
![Page 2: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/2.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Contents
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
2 / 77
![Page 3: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/3.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Smartphones are becoming more and more essential tohumans.
I As of 2016, 3.9 billion smartphone subscriptions(expected to reach 6.8 billion in 2022) out of 7.5 billionmobile phone subscriptions in the world [1].
I Yet, interaction with smartphones is largely bound totheir screens (limited by screen size, battery power andcomputation capability).
3 / 77
![Page 4: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/4.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Smartphones are becoming more and more essential tohumans.
I As of 2016, 3.9 billion smartphone subscriptions(expected to reach 6.8 billion in 2022) out of 7.5 billionmobile phone subscriptions in the world [1].
I Yet, interaction with smartphones is largely bound totheir screens (limited by screen size, battery power andcomputation capability).
3 / 77
![Page 5: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/5.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:I Camera based systems [2]. Limited camera field of
view, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
![Page 6: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/6.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:
I Camera based systems [2]. Limited camera field ofview, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
![Page 7: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/7.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:I Camera based systems [2]. Limited camera field of
view, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
![Page 8: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/8.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:I Camera based systems [2]. Limited camera field of
view, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
![Page 9: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/9.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:I Camera based systems [2]. Limited camera field of
view, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
![Page 10: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/10.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
RF based gesture recognition
Some approaches tries to introduce new HW into thesmartphones:
I Google’s Soli project [4].
I Specialized gesture recognition radar chip.
Other approaches leverage the existing phone capabilities:
I Sense activity and gestures using FM,GSM/WCDM/LTE or Wi-Fi signals [5], [6], [7] and [8].
5 / 77
![Page 11: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/11.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
RF based gesture recognition
Some approaches tries to introduce new HW into thesmartphones:
I Google’s Soli project [4].
I Specialized gesture recognition radar chip.
Other approaches leverage the existing phone capabilities:
I Sense activity and gestures using FM,GSM/WCDM/LTE or Wi-Fi signals [5], [6], [7] and [8].
5 / 77
![Page 12: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/12.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
RF based gesture recognition
Advantages:
I Require no line of sight between the gesture subject andthe smartphone.
I Consume less power.
I Ubiquitous.
6 / 77
![Page 13: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/13.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Radio wave propagation
Static or moving objects (e.g. a human hand) impact thesignal power at the receiving end.
7 / 77
![Page 14: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/14.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI
8 / 77
![Page 15: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/15.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI
9 / 77
![Page 16: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/16.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI
10 / 77
![Page 17: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/17.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI
11 / 77
![Page 18: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/18.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI
12 / 77
![Page 19: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/19.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI
13 / 77
![Page 20: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/20.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI
14 / 77
![Page 21: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/21.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI
15 / 77
![Page 22: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/22.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI
16 / 77
![Page 23: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/23.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Sample RSSI measurements (typing in akeyboard)
17 / 77
![Page 24: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/24.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Sample RSSI measurements (walking)
18 / 77
![Page 25: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/25.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Sample RSSI measurements (performing Swipegesture)
19 / 77
![Page 26: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/26.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Hand gesture recognition from Wi-Fi RSSI
20 / 77
![Page 27: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/27.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Hand gesture recognition from Wi-Fi RSSI
21 / 77
![Page 28: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/28.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Hand gesture recognition from Wi-Fi RSSI
Prediction: no gesture (or Noise)
22 / 77
![Page 29: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/29.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Hand gesture recognition from Wi-Fi RSSI
Prediction: Swipe
23 / 77
![Page 30: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/30.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Hand gesture recognition from Wi-Fi RSS
Prediction: Swipe
24 / 77
![Page 31: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/31.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Hand gesture recognition from Wi-Fi RSSI
Prediction: Noise
25 / 77
![Page 32: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/32.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Hand gesture recognition from Wi-Fi RSSI
Prediction: Push
26 / 77
![Page 33: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/33.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Hand gesture recognition from Wi-Fi RSSI
Prediction: Noise
27 / 77
![Page 34: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/34.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Challenges
28 / 77
![Page 35: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/35.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Challenges
I Hand gestures and other background activities (e.g.walking) have closely similar impacts on the Wi-Fisignal.
I Wi-Fi RSSI stream is bursty (occurs in short non regularepisodes).
29 / 77
![Page 36: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/36.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Challenges
I Hand gestures and other background activities (e.g.walking) have closely similar impacts on the Wi-Fisignal.
I Wi-Fi RSSI stream is bursty (occurs in short non regularepisodes).
29 / 77
![Page 37: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/37.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI stream is bursty
Wi-Fi frames received by a smartphone while browsingFacebook.
30 / 77
![Page 38: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/38.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI stream is bursty
Wi-Fi frames received by a smartphone while playing aYoutube video.
31 / 77
![Page 39: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/39.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Previous work
32 / 77
![Page 40: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/40.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Previous work
I Wi-Fi RSSI was used to recognize activities (e.g.walking) on smartphones [9].
I It was also used to recognize moving hand gestures onsmartphones [8], [10] and [11].
I But, to gain access to enough RSSI samples:I The Wi-Fi interface has to operate on the monitor
mode (which prevents other applications from usingthe Wi-Fi interface).
I A rooted Android OS was needed, to install a specialWi-Fi firmware.
I Supported by a limited subset of the Wi-Fi devices.
33 / 77
![Page 41: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/41.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Previous work
I Wi-Fi RSSI was used to recognize activities (e.g.walking) on smartphones [9].
I It was also used to recognize moving hand gestures onsmartphones [8], [10] and [11].
I But, to gain access to enough RSSI samples:I The Wi-Fi interface has to operate on the monitor
mode (which prevents other applications from usingthe Wi-Fi interface).
I A rooted Android OS was needed, to install a specialWi-Fi firmware.
I Supported by a limited subset of the Wi-Fi devices.
33 / 77
![Page 42: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/42.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Previous work
I Wi-Fi RSSI was used to recognize activities (e.g.walking) on smartphones [9].
I It was also used to recognize moving hand gestures onsmartphones [8], [10] and [11].
I But, to gain access to enough RSSI samples:I The Wi-Fi interface has to operate on the monitor
mode (which prevents other applications from usingthe Wi-Fi interface).
I A rooted Android OS was needed, to install a specialWi-Fi firmware.
I Supported by a limited subset of the Wi-Fi devices.
33 / 77
![Page 43: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/43.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Objectives
34 / 77
![Page 44: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/44.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Project objectives
Demonstrate the possibility to recognize dynamic handgestures on smartphones from the Wi-Fi RSSI stream,without modification, in a passive online setting.
I dynamic: involves hand movement.
I passive: leverages existing Wi-Fi sources.
I online: in realtime on the smartphone.
I without modification: without requiring additionalHW, or core SW modification.
35 / 77
![Page 45: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/45.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
36 / 77
![Page 46: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/46.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:I Suitable for sequential inputs (e.g. audio and video
signals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
![Page 47: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/47.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:I Suitable for sequential inputs (e.g. audio and video
signals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
![Page 48: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/48.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:
I Suitable for sequential inputs (e.g. audio and videosignals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
![Page 49: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/49.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:I Suitable for sequential inputs (e.g. audio and video
signals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
![Page 50: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/50.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:I Suitable for sequential inputs (e.g. audio and video
signals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
![Page 51: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/51.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Traffic induction
38 / 77
![Page 52: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/52.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Traffic induction
39 / 77
![Page 53: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/53.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Traffic induction
40 / 77
![Page 54: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/54.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
LSTM RNN model
I N = 200 neurons/LSTM cell
I T = 50 (RNN time steps)
41 / 77
![Page 55: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/55.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Recognition system diagram
42 / 77
![Page 56: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/56.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Experiments
43 / 77
![Page 57: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/57.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Performed hand gestures:
44 / 77
![Page 58: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/58.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Swipe samples collection session:
45 / 77
![Page 59: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/59.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Sample Swipe gestures:
46 / 77
![Page 60: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/60.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Sample Push gestures:
47 / 77
![Page 61: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/61.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Sample Pull gestures:
48 / 77
![Page 62: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/62.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Spatial setup:
49 / 77
![Page 63: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/63.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
I Summary of the collected dataset:
Dataset 1 2 3 4
Location room room room two roomsInduction
√ √ √
Internet√
Size 440 432 434 337
I An Android app was developed to collect the dataset.
50 / 77
![Page 64: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/64.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Training and Evaluation of the LSTM RNN model wasconducted on a Laptop (hence offline), using the collecteddataset.
51 / 77
![Page 65: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/65.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Traffic induction impact on prediction accuracy.
Dataset 1 2 3 4
Location room room room two roomsInduction
√ √ √
Internet√
LSTM accuracy 91% 83% 78% 87%
52 / 77
![Page 66: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/66.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Swipe and Push gestures are hardly distinguishable wheninduction is OFF.
Left: Swipe gesture with induction ON. Middle and Right:Swipe and Push gestures respectively performed while notraffic is OFF.
53 / 77
![Page 67: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/67.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Prediction accuracy as a function of the number of samplesper prediction window.
10 20 30 40 50 80 100
85
888889
91
89
83
samples per window
accu
racy
54 / 77
![Page 68: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/68.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Prediction accuracy as a function of the number of thehidden (LSTM) layers.
1 2 3
9192
90
number of layers
accu
racy
55 / 77
![Page 69: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/69.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
The LSTM RNN model accuracy when trained with fractionsof (Dataset1 + Dataset2 + Dataset4).
0.1 0.25 0.5 1
83
88
9294
dataset fraction
accu
racy
56 / 77
![Page 70: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/70.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Online experiments
Evaluating the full solution implementation (an Androidapp) on a smartphone.
57 / 77
![Page 71: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/71.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Online experiments
Spatial setup
58 / 77
![Page 72: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/72.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Online experiments
I Accuracy of Line-of-sight (LOS) experiments is ∼81%.
I Accuracy of no Line-of-sight (No LOS) experiments is∼74%.
I The Overall accuracy is ∼78%.
59 / 77
![Page 73: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/73.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Online experiments
When no hand gesture is performed over a period of thirtyminutes, the false positivie rate was ∼8%.
Gesture number of predictions (%)
Noise (correct prediction) 1652 (92.1%)Swipe (False positive) 61 (3.4%)Push (False positive) 62 (3.5%)Swipe (False positive) 18 (1.0%)
60 / 77
![Page 74: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/74.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
DemoClick here - Takes you to YouTube!
61 / 77
![Page 75: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/75.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Discussion
62 / 77
![Page 76: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/76.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Conclusion
I We demonstrated its possible to predict hand gestureson unmodified smartphones from Wi-Fi RSSI.
I The recognition accuracy can be improved by collectingmore data, and increasing the model size.
I The recognition accuracy can be improved by samplingRSSI at higher frequency.
63 / 77
![Page 77: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/77.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Conclusion
I We demonstrated its possible to predict hand gestureson unmodified smartphones from Wi-Fi RSSI.
I The recognition accuracy can be improved by collectingmore data, and increasing the model size.
I The recognition accuracy can be improved by samplingRSSI at higher frequency.
63 / 77
![Page 78: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/78.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Conclusion
I We demonstrated its possible to predict hand gestureson unmodified smartphones from Wi-Fi RSSI.
I The recognition accuracy can be improved by collectingmore data, and increasing the model size.
I The recognition accuracy can be improved by samplingRSSI at higher frequency.
63 / 77
![Page 79: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/79.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Limitations
I Vulnerability to interference from background activities.
I High CPU usage (25%).
64 / 77
![Page 80: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/80.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Limitations
I Vulnerability to interference from background activities.
I High CPU usage (25%).
64 / 77
![Page 81: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/81.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Publication
Published in IEEE Sensors (2019).Citation: Haseeb MA, Parasuraman R. Wisture:Touch-Less Hand Gesture Classification in UnmodifiedSmartphones Using Wi-Fi Signals. IEEE SensorsJournal. 2018 Oct 16;19(1):257-67.
65 / 77
![Page 82: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/82.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Publication
We open-sourced most of the codes and dataset (datacollection and Wisture recognition).https://github.com/mohaseeb/wisturehttps://www.ieee-dataport.org/documents/wi-fi-signal-strength-measurements-smartphone-various-hand-gestures
66 / 77
![Page 83: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/83.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be:
hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
![Page 84: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/84.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise
and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
![Page 85: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/85.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
![Page 86: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/86.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
![Page 87: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/87.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
![Page 88: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/88.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
![Page 89: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/89.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
![Page 90: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/90.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Native support on Wi-Fi devices for a cheap high frequencysampling of Wi-Fi RSS.
I By inducing traffic at the Wi-Fi device level, the OS isbypassed, which results in a higher throughput at areduced power level.
I Reliable recognition capability at lower cost, comparedto, for example, introducing a completely new HW likeGoogle’s Soli [4].
68 / 77
![Page 91: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/91.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Native support on Wi-Fi devices for a cheap high frequencysampling of Wi-Fi RSS.
I By inducing traffic at the Wi-Fi device level, the OS isbypassed, which results in a higher throughput at areduced power level.
I Reliable recognition capability at lower cost, comparedto, for example, introducing a completely new HW likeGoogle’s Soli [4].
68 / 77
![Page 92: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/92.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Native support on Wi-Fi devices for a cheap high frequencysampling of Wi-Fi RSS.
I By inducing traffic at the Wi-Fi device level, the OS isbypassed, which results in a higher throughput at areduced power level.
I Reliable recognition capability at lower cost, comparedto, for example, introducing a completely new HW likeGoogle’s Soli [4].
68 / 77
![Page 93: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/93.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Questions
69 / 77
![Page 94: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/94.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References I
Ericsson mobility report.https://www.ericsson.com/en/mobility-report/latest-mobile-statistics.
[Online; accessed 22-Oct-2019].
Jie Song, Gabor Soros, Fabrizio Pece, Sean RyanFanello, Shahram Izadi, Cem Keskin, and OtmarHilliges.In-air gestures around unmodified mobile devices.In Proceedings of the 27th Annual ACM Symposium onUser Interface Software and Technology, UIST ’14,pages 319–329, New York, NY, USA, 2014. ACM.
70 / 77
![Page 95: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/95.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References II
Gabe Cohn, Daniel Morris, Shwetak Patel, and DesneyTan.Humantenna: Using the body as an antenna forreal-time whole-body interaction.In Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems, CHI ’12, pages1901–1910, New York, NY, USA, 2012. ACM.
Project soli.https://atap.google.com/soli/, 2016.[Online; accessed 1-June-2016].
71 / 77
![Page 96: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/96.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References III
Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi,Shwetak Patel, and Matthew S. Reynolds.Sideswipe: Detecting in-air gestures around mobiledevices using actual gsm signal.In Proceedings of the 27th Annual ACM Symposium onUser Interface Software and Technology, UIST ’14,pages 527–534, New York, NY, USA, 2014. ACM.
Stephan Sigg, Shuyu Shi, Felix Buesching, Yusheng Ji,and Lars Wolf.Leveraging rf-channel fluctuation for activity recognition:Active and passive systems, continuous and rssi-basedsignal features.In Proceedings of International Conference on Advancesin Mobile Computing & Multimedia, MoMM ’13,pages 43:43–43:52, New York, NY, USA, 2013. ACM.
72 / 77
![Page 97: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/97.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References IV
Rajalakshmi Nandakumar, Bryce Kellogg, andShyamnath Gollakota.Wi-fi gesture recognition on existing devices.CoRR, abs/1411.5394, 2014.
S. Sigg, U. Blanke, and G. Troster.The telepathic phone: Frictionless activity recognitionfrom wifi-rssi.In Pervasive Computing and Communications (PerCom),2014 IEEE International Conference on, pages 148–155,2014.
73 / 77
![Page 98: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/98.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References V
Stephan Sigg, Mario Hock, Markus Scholz, GerhardTroster, Lars Wolf, Yusheng Ji, and Michael Beigl.Mobile and Ubiquitous Systems: Computing,Networking, and Services: 10th InternationalConference, MOBIQUITOUS 2013, Tokyo, Japan,December 2-4, 2013, Revised Selected Papers, chapterPassive, Device-Free Recognition on Your Mobile Phone:Tools, Features and a Case Study, pages 435–446.Springer International Publishing, Cham, 2014.
Christoph Rauterberg, Mathias Velten, Stephan Sigg,and Xiaoming Fu.Simply use the force - implementation of rf-basedgesture interaction on an android phone.In IEEE/KuVS NetSys 2015 adjunct proceedings, 2015.
74 / 77
![Page 99: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/99.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References VI
C. Rauterberg and X. Fu.Demo abstract: Use the force, luke: Implementation ofrf-based gesture interaction on an android phone.In Pervasive Computing and Communication Workshops(PerCom Workshops), 2015 IEEE InternationalConference on, pages 190–192, March 2015.
Anthony Bagnall, Aaron Bostrom, James Large, andJason Lines.The great time series classification bake off: Anexperimental evaluation of recently proposed algorithms.extended version.CoRR, abs/1602.01711, 2016.
75 / 77
![Page 100: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/100.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Supporting slides
76 / 77
![Page 101: Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones · 2020-01-23 · I As of 2016, 3.9 billion smartphone subscriptions (expected to reach 6.8 billion in 2022) out of](https://reader034.fdocuments.us/reader034/viewer/2022050602/5fa9a5582a71e16ed10823e2/html5/thumbnails/101.jpg)
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Prediction accuracies, training and prediction times fordifferent algorithms evaluated using Dataset1.
Algorithm Accuracy Sample prediction time (ms)
K-NN DTW 90% (±28) 964.15FS 85% (±4.6) 0.01STE 91% (±1.1) 26.86LTS 93% (±2.3) 9.29EE 93% (±1.7) 23.09COTE 94% (±2.4) 178.20LSTM RNN 91% (±3.1) 7.04
STE, EE and COTE are ensemble methods that arecomputationally heavy [12].
77 / 77