We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong...
-
Upload
andres-hogan -
Category
Documents
-
view
219 -
download
0
Transcript of We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong...
![Page 1: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/1.jpg)
We Can Hear You with Wi-Fi !
Guanhua Wang
Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni
Hong Kong University of Science and Technology
![Page 2: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/2.jpg)
Advanced Research in ISM band
• Localization
![Page 3: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/3.jpg)
Advanced Research in ISM band
•Gesture Recognition
![Page 4: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/4.jpg)
Advanced Research in ISM band
•Object Detection
Full Duplex Backscatter. In hotnets 2013
![Page 5: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/5.jpg)
• Localization
•Gesture recognition
•Object Classification
They enable Wi-Fi to “SEE” target objects.
Advanced Research in ISM band
![Page 6: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/6.jpg)
Can we enable Wi-Fi signals to HEAR talks?
![Page 7: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/7.jpg)
Can we enable Wi-Fi signals to HEAR talks?
![Page 8: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/8.jpg)
What is WiHear?
![Page 9: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/9.jpg)
“Hearing” human talks with Wi-Fi signals
Hello
Non-invasive and device-free
![Page 10: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/10.jpg)
Hearing through walls and doors
I am upset.
Understanding complicated human behavior (e.g.
mood)
![Page 11: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/11.jpg)
Hearing multiple people simultaneously
MIMO Technology
Easy to be implemented in commercial Wi-Fi products
![Page 12: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/12.jpg)
How does WiHear work?
![Page 13: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/13.jpg)
WiHear Framework
Feature Extraction
Classification & Error Correction
Learning-based Lip Reading
LaptopLaptop
PeoplePeople
APAP
MIMO Beamforming
Vows and consonants Filtering
Wavelet Transform
Remove Noise
Mouth Motion Profiling
Partial Multipath Removal
Segmentation
ProfileBuilding
![Page 14: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/14.jpg)
Mouth Motion Profiling
• Locating on Mouth
• Filtering Out-Band Interference
• Partial Multipath Removal
•Mouth Motion Profile Construction
• Discrete Wavelet Packet Decomposition
![Page 15: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/15.jpg)
Mouth Motion Profiling
• Locating on Mouth
• Filtering Out-Band Interference
• Partial Multipath Removal
•Mouth Motion Profile Construction
• Discrete Wavelet Packet Decomposition
![Page 16: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/16.jpg)
Locating on Mouth
APAP
PeoplePeople
LaptopLaptop
T1
T2T3
T3
![Page 17: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/17.jpg)
Mouth Motion Profiling
• Locating on Mouth
• Filtering Out-Band Interference
• Partial Multipath Removal
•Mouth Motion Profile Construction
• Discrete Wavelet Packet Decomposition
![Page 18: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/18.jpg)
Filtering Out-Band Interference
•Signal changes caused by mouth motion: 2-5 Hz
•Adopt a 3-order Butterworth IIR band-pass filterCancel the DC componentCancel wink issue (<1 Hz)
Cancel high frequency interference
The impact of wink (as denoted in the dashed red box).
![Page 19: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/19.jpg)
Mouth Motion Profiling
• Locating on Mouth
• Filtering Out-Band Interference
• Partial Multipath Removal
•Mouth Motion Profile Construction
• Discrete Wavelet Packet Decomposition
![Page 20: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/20.jpg)
Partial Multipath Removal
•Mouth movement: Non-rigid
• Covert CSI (Channel State Information) from frequency domain to time domain via IFFT
•Multipath removal threshold: >500 ns
• Convert processed CSI (with multipath < 500ns) back to frequency domain via FFT
The multipath threshold value can be adjusted to achieve better performance
![Page 21: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/21.jpg)
Mouth Motion Profiling
• Locating on Mouth
• Filtering Out-Band Interference
• Partial Multipath Removal
•Mouth Motion Profile Construction
• Discrete Wavelet Packet Decomposition
![Page 22: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/22.jpg)
Mouth Motion Profiling
• Locating on Mouth
• Filtering Out-Band Interference
• Partial Multipath Removal
•Mouth Motion Profile Construction
• Discrete Wavelet Packet Decomposition
![Page 23: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/23.jpg)
Discrete Wavelet Packet Decomposition• A Symlet wavelet filter of order 4 is selected
![Page 24: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/24.jpg)
WiHear Framework
Feature Extraction
Classification & Error Correction
Learning-based Lip Reading
LaptopLaptop
PeoplePeople
APAP
MIMO Beamforming
Vows and consonants Filtering
Wavelet Transform
Remove Noise
Mouth Motion Profiling
Partial Multipath Removal
Segmentation
ProfileBuilding
![Page 25: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/25.jpg)
Lip Reading
• Segmentation
• Feature Extraction
• Classification
• Context-based Error Correction
![Page 26: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/26.jpg)
Lip Reading
• Segmentation
• Feature Extraction
• Classification
• Context-based Error Correction
![Page 27: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/27.jpg)
Segmentation
• Inter word segmentationSilent interval between words
• Inner word segmentation Words are divided into phonetic events
![Page 28: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/28.jpg)
Lip Reading
• Segmentation
• Feature Extraction
• Classification
• Context-based Error Correction
![Page 29: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/29.jpg)
Feature Extraction• Multi-Cluster/Class Feature Selection (MCFS) scheme
![Page 30: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/30.jpg)
Lip Reading
• Segmentation
• Feature Extraction
• Classification
• Context-based Error Correction
![Page 31: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/31.jpg)
Lip Reading
• Segmentation
• Feature Extraction
• Classification
• Context-based Error Correction
![Page 32: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/32.jpg)
Extending To Multiple Targets
•MIMO: Spatial diversity via multiple Rx antennas
• ZigZag decoding: a single Rx antenna
![Page 33: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/33.jpg)
Implementation
Floor plan of the testing environment. Experimental scenarios layouts. (a) line of sight; (b) non-line-of-sight; (c) through wall Tx side; (d) through wall Rx side; (e) multiple Rx; (f) multiple link pairs.
![Page 34: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/34.jpg)
Vocabulary
• Syllables: [æ], [e], [i], [u], [s], [l], [m], [h], [v], [ɔ], [w], [b], [j], [ ʃ ].
•Words: see, good, how, are, you, fine, look, open, is, the, door,
thank, boy, any, show, dog, bird, cat, zoo, yes, meet, some, watch, horse, sing, play, dance, lady, ride, today, like, he, she.
![Page 35: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/35.jpg)
Automatic Segmentation Accuracy
Automatic segmentation accuracy for (a) Inner-word segmentation
on commercial devices (b) Inter-word segmentation on commercial devices (c) Inner-word segmentation on USRP(d) Inter-word segmentation on USRP
![Page 36: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/36.jpg)
Classification Accuracy
![Page 37: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/37.jpg)
Training Overhead
![Page 38: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/38.jpg)
Impact of Context-based Error Correction
![Page 39: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/39.jpg)
Performance with Multiple Receivers
Example of different views for pronouncing words
![Page 40: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/40.jpg)
Performance for Multiple Targets
Performance of multiple users with multiple link pairs.
Performance of zigzag decoding for multiple users.
![Page 41: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/41.jpg)
Through Wall Performance
Performance of two through wall scenarios. Performance of through wall with multiple Rx.
![Page 42: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/42.jpg)
Resistance to Environmental Dynamics
Waveform of a 4-word sentence without interference of ISM band signals or irrelevant human motions
Impact of irrelevant human movements interference
Impact of ISM band interference
![Page 43: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/43.jpg)
Conclusion
• WiHear is the 1st prototype in the world, trying to use Wi-Fi signal to sense and recognize human talks.
• WiHear takes the 1st step to bridge communication between human speaking and wireless signals.
• WiHear introduces a new way so that machine can sense more complicated human behaviors (e.g. mood).
![Page 44: We Can Hear You with Wi-Fi ! Guanhua Wang Yongpan Zou, Zimu Zhou, Kaishun Wu, Lionel M. Ni Hong Kong University of Science and Technology.](https://reader035.fdocuments.us/reader035/viewer/2022062307/551c1668550346a34f8b5671/html5/thumbnails/44.jpg)
Thank you for your listening !