Robust Device-Free Indoor Localization

When Machine Learning Meets Wi-Fi

Steve Liu, VP R&D, Chief-Scientist

Samsung AI Center – Montreal &

Professor & William Dawson Scholar

McGill University

Nov. 5th, 2019

Disclaimer: Any views or opinions presented in this talk are personal and do not represent those of people, institutions, organizations that the presenter may or may not be associated with in professional or personal capacity.

Outline

Robust Device-Free Indoor Localization

• Indoor Localization & Background

– Fingerprinting-based Device-Free Localization Approach

• A Major Challenge: Environment Changes

• AutoFi

• Experiments

• More Recent Advances: Toward Robustness in Real World

• Conclusions & Future Work Directions

3

Indoor Localization & Background

Why this Topic?

The next big thing!

• Internet of Things

• AI and Machine Leaning (=> Intelligent)

• Advanced Communications like Wi-Fi 6 & 5G (=> Connected )

5

Indoor Localization

GPS signal not available

6

Applications

What can we do with location information?

Indoor navigation

Smart home automation

Automated industry

7

State-of-the-Art

Device-Oriented

• Pro: High accuracy: 10cm-level (e.g., [1],[2],[3])

• Con: Complicated design– Customized hardware

– Antenna arrays (10+ antenna)

– Usually heavy computational overhead

• Locating devices, NOT users

[1] M. Kotaru, K. R. Joshi, D. Bharadia, and S. Katti, “Spotfi: Decimeter level localization using WiFi”, Sigcomm ‘15. [2] X. Li, D. Zhang, Q. Lv, J. Xiong, S. Li, Y. Zhang, and H. Mei, “Indotrack: Device-free indoor human tracking with commodity WiFi,” IMWUT’ 17[3] K. Qian, C. Wu, Y. Zhang, G. Zhang, Z. Yang, and Y. Liu, “Widar2.0: Passive human tracking with a single WiFi link,” MobiSys ‘18.

8

Why Device-free Localization?

People are not always carrying devices

• Cannot, or Do not want to carry

9

State of the Art

Device-Oriented

Device-Free

• Triangulation methods– Utilize user’s blocking/reflection of the signals

– Calculating Time-of-Fight (ToF), Angle-of-Arrival (AoA), Propagation features, etc.

– Dedicated devices required

10

Disadvantages of Dedicated Devices

• Cost– Equipment cost

– Deployment

– Triangulation: positions of devices must be known in advance

• Scalability

• Range limitation (motion sensors, RFID, …)

• Blocked by walls

• …

Motion Sensors CamerasRFID

Tag & Reader

11

State of the Art

Device-Oriented

Device-Free

• Triangulation methods

• Fingerprinting-based method– Correlating environment features with human locations

– Environment features:

• Wireless signals, e.g., Wi-Fi

• Light or sound

• Magnetic field

• …

12

Benefits of Wi-Fi based DfP Localization

Work through walls!

Available Everywhere!

13

Wi-Fi Based DfP Localization

Fingerprinting: Associate Wi-Fi features such as CSI (Channel State Information) or

RSSI (Received Signal Strength Indication) with users’ locations.

14

Wi-Fi Channel State Information

Channel State Information (CSI)

• Describe the Wi-Fi channel properties

• Sensitive to human locations

• Provides much more information than RSSI

• Readily available from commercial Wi-Fi cards

15

1 RSSI → 1 integer

56 subcarriers with CSI values(complex numbers) → 56 magnitudes + 56 phases

→ 112 floating point numbers

More Background on Wi-Fi CSI

Wi-Fi Multiple-Input Multiple-Output (MIMO)

e.g., Intel 5300 NICe.g., Linksys WRT160N16

More Background on Wi-Fi CSI

MIMO in math

Received signals

Streams (Spatial Channels)CSI

Transmittedsignals

Noise

Source: http://www.sharetechnote.com/html/Communication_ChannelModel.html

𝒚 = 𝑯 𝒙 + 𝒏

17

How does CSI look like?

CSI profiles at h11

• Captures the math representation of channel characteristics from transmitter antenna 1 to receiver antenna 1

• Time/frequency variant

Time (seconds)

Frequency(Subcarriers)

CSIMagnitude

18

CSI Profiles are Location-Dependent

Subcarriers

CSI

Magnitude

19

CSI Profiles to CSI Fingerprints

A CSI fingerprint of location L is a CSI profile associated with this location

• i.e., fingerprint = CSI profile at location L

20

How to use ML for Localization?

Use CSI fingerprints to train ML classifiers

• Locations – labels (output), CSI profiles (input)

• SVM, Random Forest, KNN, Neural Networks, etc.

MLClassifiers

For training

21

Classification for Localization

Trained models to classify the labels of new (online measured) CSI profiles

• Predicted labels = estimated locations

Trained ML

Classifiers

10 locations, (sub-)meter-level resolution - 99.7% accuracy.22

A Major Challenge

- Environment Changes

A Major Challenge

The changes in the indoor environment make the (old) fingerprints inconsistent with

the current situation (new).

24

A Major Challenge

The CSI fingerprints will be “contaminated” by environment changes.

• The recorded fingerprints (old) no longer represent the changed environment (new)

25

A Major Challenge

The CSI fingerprints will be “contaminated” by environment changes.

• The recorded fingerprints (old) no longer represent the changed environment (new)

Before After

26

A Major Challenge

Localization accuracy drops significantly

• A case study:

Record the fingerprints again, and retrain the model?

27

Impractical: Inconvenient & time-consuming

AutoFi

Idea: Can We Reuse the Fingerprints?

Reconstruct vectors measured from the changed domain to the old fingerprint domain, so we can reuse the already trained model

𝑀𝑛𝑒𝑤−𝑜𝑙𝑑

-------------------------------------------------------------------------------------

New vector domain(Contaminated CSI profiles)

Old vector domain(fingerprints)

𝑀𝑛𝑒𝑤−𝑜𝑙𝑑

𝑉𝑒𝑚𝑝𝑡𝑦𝑜𝑙𝑑 𝑉𝑝1

𝑜𝑙𝑑

𝑉𝑒𝑚𝑝𝑡𝑦𝑛𝑒𝑤 𝑉𝑝1

𝑛𝑒𝑤

29

The Mapping Functions

Map CSI from the new domain to the old domain

Contaminated CSI profiles(locations unknown)

Unknown mapping How to determine?

𝑉𝑝1𝑜𝑙𝑑 = 𝑀𝑝1 × 𝑉𝑝1

𝑛𝑒𝑤 + 𝑛𝑝1,

𝑉𝑝2𝑜𝑙𝑑 = 𝑀𝑝2 × 𝑉𝑝2

𝑛𝑒𝑤 + 𝑛𝑝2 ,

⋮

𝑉𝑝𝑁𝑜𝑙𝑑 = 𝑀𝑝𝑁 × 𝑉𝑝𝑁

𝑛𝑒𝑤 + 𝑛𝑝𝑁 .

Fingerprints in Old Domain(associated with locations)

30

The Mapping Functions

Approximation: the mapping functions are the same for different locations

• Domain-to-domain mapping

𝑀𝑝1 = 𝑀𝑝2 = ⋯ = 𝑀𝑝𝑁 = 𝑀.

Contaminated domain (New) Fingerprint domain (Old)

M: New →Old

31

The Mapping Function

Mapping function: M

𝑉𝑝1𝑜𝑙𝑑 = 𝑀 × 𝑉𝑝1

𝑛𝑒𝑤 + 𝑛𝑝1,

𝑉𝑝2𝑜𝑙𝑑 = 𝑀 × 𝑉𝑝2

𝑛𝑒𝑤 + 𝑛𝑝2 ,

⋮

𝑉𝑝𝑁𝑜𝑙𝑑 = 𝑀 × 𝑉𝑝𝑁

𝑛𝑒𝑤 + 𝑛𝑝𝑁 .

32

Fingerprints in Old Domain(associated with locations)

Contaminated CSI profiles(locations unknown)

The Mapping Functions

How to automatically determine the mapping function M?

𝑉?𝑜𝑙𝑑 = 𝑀 × 𝑉?

𝑛𝑒𝑤 + 𝑛?

Both unknown!

• In order to get M, we need to automatically detect a location p as a

reference point such that both 𝑉𝑝𝑛𝑒𝑤𝑎𝑛𝑑 𝑉𝑝

𝑜𝑙𝑑are known

33

Determine M

Observation: The status of an empty area can be detected using a rule-based algorithm

• Reference point (state) = Detecting Empty

• Both before and after the environmental changes

• How? Using variance of CSI magnitude (can be detected automatically)

Associate 𝑉?𝑛𝑒𝑤 with ?=Empty => 𝑉𝑒𝑚𝑝𝑡𝑦

𝑜𝑙𝑑 = 𝑀 × 𝑉𝑒𝑚𝑝𝑡𝑦𝑛𝑒𝑤 + 𝑛

34

Contaminant Removal

𝑀 = 𝑀E𝑚𝑝𝑡𝑦

Contaminated profile (raw CSI measured in the newly changed environment)

𝑀

? ?

De-contaminated (purified) profile• Transformed CSI• In the old domain, i.e. when

there is no environ changes

Trained ML model

Determine Location

35

Contaminant Removal Example

An example of contaminant removal result @ P1

Still some residual errors!

36

Autoencoder

A neural network learns efficient data representation (encoding) of the input data

• Encoder

• Decoder

– To (re)generate data

37

Training the Autoencoder

The autoencoder tries to reconstruct the input

• Training: Inputs and outputs are the same – the fingerprint profiles recorded

• Learns an efficient representation in the recorded domain (fingerprint profiles)

• Use BP to train

Fingerprint profiles

The encoding

Fingerprint profiles

38

Denoising with the Autoencoder (Inference)

• The autoencoder denoises measured new CSI profiles

– Using the (already) purified profiles as input

– Old fingerprint domain features are identified through the coding/decoding

– Features related to the environment changes are diluted/omitted

New (purified) profiles Reconstructed profiles

The encoding

39

Putting All Components Together

The architecture of AutoFi (Training in red, localization in blue).

40

Experiments

Experiment Setup

A Linksys WRT160N router, a laptop with Intel 5300 NIC.

• Wi-Fi traffic: 10 – 20 pings per second.

• Meter-level resolution (minimum distance 80cm)

42

Experiment Setup

Introducing the “contaminants”

43

Experiment Results

Baseline, no mapping was applied, Random Forest (RF).

Opening windows Opening doors

Accuracy Mean Min Mean Min

Baseline 18.8% 0% 41.7% 0%

44

Experiment Results

Using only contaminant removal technique

Opening windows Opening doors

Accuracy Mean Min Mean Min

Baseline 18.8% 0% 41.7% 0%

Contaminant Rmv 69.0% 0% 70.0% 0%

45

Experiment Results

AutoFi (using both contaminant removal and autoencoder)

Opening windows Opening doors

Accuracy Mean Min Mean Min

Baseline 18.8% 0% 41.7% 0%

Contaminant Rmv 69.0% 0% 70.0% 0%

AutoFi 84.9% 47.6% 90.2% 71.3% 46

Brief Summary

• Problem for robust localization

– Small variations in the environment may significantly contaminate the

fingerprints

• Solution - AutoFi

– Reuse the fingerprints and the trained ML model with a contaminant

removal technique

automatically maps the contaminated profiles back to the fingerprint

domain

– Utilize an autoencoder to further denoise the purified profiles

47

More Recent Advances

- Toward Robustness in Real World

More Robust Motion Detection

AutoFi used CSI variance as an indicator of human motion

• CSI variance is not a robust indicator of human motion

• In some empty rooms, CSI variance caused by noise can be even higher than that caused by human motion

• Simply applying white noise filtering does not work

– Because this also removes the variance caused by human motion

Our solution MoFi [to appear in RTSS@Work 2019]

49

More Robust Motion Detection

• Robustness Study on Motion Detection

– Different users

– Different rooms

– Different AP/device placements

– Environment changes, e.g., moved furniture, open/close doors/windows

– Interference from other 2.4GHz devices, e.g., microwave ovens and Bluetooth beacons

50

Addressing User Diversity

Different body shapes yield different CSI profiles

• A system trained on one user may not work for a new user

• Labelling CSI profiles for a new user is mostly impractical

– Requiring user involvement

– Time consuming

• Work under submission and review

51

Wi-Fi Sensing

A new IEEE TIG (Topic Interest Group) on Wi-Fi sensing:

52

Wi-Fi based Motion Detection

Photo credit: Linksys: https://www.linksys.com/us/linksys-aware/ 53

Conclusions & Future Directions

Conclusions and Future Directions

• Wi-Fi based device-free localization is very promising

• Robust ML solutions have been developed for real-life scenarios

– Robust to environment changes: AutoFi

– Robust to random noises: MoFi

– Robust to user diversity: Under submission

• Future: Many more to explore!

– Multi-user localization

– Fast bootstrap/adaptation

– …

55

BACKUP SLIDES

57

Physics Behind This

Signals affected by human body

58

Physics Behind This

Different body locations introduce different effects

59