Spectral Tweets: Dynamic Spectrum Sensing for Android Devices

Advisor: Nikos Sidiropoulos

Team Members:

Martin Corbett, Michael Fiore, Alex Hulke, Grayson Malinowski, Michael Owczarek

Submission Date: 12/17/2012

2

Table of Contents

Table of Contents ...........................................................................................................................2

Executive summary ........................................................................................................................3

Problem Definition .........................................................................................................................4

Background & Key Concepts ..............................................................................................4

Customer Needs ...................................................................................................................6

Product Design Specifications .............................................................................................6

Concept Design ....................................................................................................................6

Design Description .........................................................................................................................7

Android Application ............................................................................................................7

PC Client ..............................................................................................................................9

Detailed Description ..............................................................................................10

Assumptions & Simplifications .............................................................................10

Matlab Script ......................................................................................................................11

Design Evaluation ........................................................................................................................12

Android Application ..........................................................................................................12

PC Client ............................................................................................................................13

Matlab Script ......................................................................................................................13

Conclusions & Recommendations ..............................................................................................14

Product Design Specification .............................................................................................14

Ideas for Improvement .......................................................................................................16

Design Process ...................................................................................................................17

Budget ................................................................................................................................17

Schedule .............................................................................................................................18

References .....................................................................................................................................19

Appendix A ...................................................................................................................................20

3

Executive Summary

Effective utilization of network resources is becoming increasingly more important as the

amount of network traffic grows. Dynamic Spectrum Sensing will prove to be an important tool

in advancing the cause of efficient network utilization. Our client needs a tool that can be used

to gather network information and process it into a useable form, which is to be utilized for

network mapping, network planning, spectrum analysis, and other applications. One of the main

goals of this project is to map wireless networks by determining the physical locations of access

points and interpolating WIFI signal strength in between them.

Our solution to our client’s needs involves three major components. The first component

is an Android application that periodically gathers GPS and WIFI information, encodes it in a

character string, and uploads it to the internet using Twitter. The second component is a

Windows program that will download the information from Twitter, parse the character string

into useful pieces of information, and save them all in a format readable by Matlab. The final

component is a Matlab script that will process the saved data and output it in a human readable

format, one possibility being a Google Maps overlay.

The current scope of this project is limited to wireless networks, but the concept could be

expanded to other types of radio signals, including cellular, Bluetooth, and others. There are two

major limitations that this project must currently deal with: communicating with GPS satellites

and maintaining an internet connection. These are both factors that would affect the

performance of the Android application.

An alternative to our solution method is to use a software radio to gather the information

and upload the information to a terminal. We chose the Android application and Twitter because

software radios are not as widely available or as portable as Android devices, both the Android

and Twitter API’s (Application Programming Interfaces) are well documented and widely used,

giving us a lot of code and information we can use to quickly get started and accomplish our

goals. For these reasons, we decided that the Android, Twitter, and Matlab solution we chose to

implement better fit our client’s needs and our relatively short timeframe.

In testing, we discovered a few bugs in the Android code, but we were able to work them

out. We also needed to tweak the Matlab code to correctly account for errors in the

approximations when we started using real data.

There are a number of improvements and advances that can be made to our finished product:

4

Expanding to other types of radio signals

Improving the user interface to the Android application and the Windows program

Adding features to the Android application and the Windows program

This project should prove to be a useful starting point from which other individuals or groups

can expand and add functionality to accomplish even more complicated and rewarding

objectives.

Problem Definition

Background & Key Concepts

An Android Device, such as a cellular phone, can be an extremely robust tool. Because

of the built in Wi-Fi functionality, it is a simple task to locate and connect to available networks.

A big part of Wi-Fi signal strength is related to its proximity to a Wi-Fi access point, but since

these access points don’t contain GPS information an end user might have to randomly relocate

themselves several times in order to obtain a decent signal strength.

Our goal is to fix this by collecting data that will be used to gauge the locations of the

Wi-Fi access points which is a first step to gaining dynamic spectrum access and sensing

capabilities. Sensing refers to the ability to be able to take readings of the current spectrum (Wi-

Fi in this particular case) in real-time. Access is all about taking that data and dynamically using

the entire spectrum in order to utilize its bandwidth effectively.

We first programmed an Android App that takes periodic measurements of Wi-Fi signal

strength across the Wi-Fi spectrum and the device’s current GPS location. The data consists of

the signal’s channel number, MAC address, and signal strength and the current GPS location.

Then the data is then encoded to fit into a single Tweet and uploaded to a designated Twitter

account. A sample tweet contains the hashtag first, then each MAC address and signal strength

pair (one character for the strength measurement and the last nine characters of the MAC

address), and finally the latitude and longitude of the phone’s location.

Next, the data is periodically downloaded and decoded by a Windows Client on the back

end which will also save the data in Matlab format. This client assumes the data encoded in the

tweet is formatted as specified above, but it also contains some error checking to make sure we

won’t get unwanted results. Key components include: a 3rd

party program, the Archivist, which

downloads the tweets and a java-based parser decodes/encodes the tweet to be readable for

5

Matlab. It is saved as a table where each row is one measurement. This table can easily be

sorted by MAC address, location, time at which it was taken, etc.

Finally, the data will be processed and converted into a dynamic spectrum sensing graph

which is the final product. Our Matlab script will work on one particular MAC address, process

all data for that address, and plot all data points with the proposed access point. The graph will

contain the location of taken measurements with a circle around it indicating that readings’

strength and the calculated location of an access point.

MatlabKeller Hall

Figure 1: Brief Overview of Design

The figure above displays our overall goal in graphical format. Following it in a

clockwise fashion, the end goal is to generate a heat map to pinpoint the location of an access

point in Keller Hall.

6

Customer Needs

# Need Importance

1 Detect Wi-Fi signal strength, MAC address, and channel numbers 5

2 Detect Wi-Fi and phone GPS locations automatically 5

3 Use Android App to collect data and upload to Twitter 5

4 Download data from Twitter to a PC for analysis 5

5 Generate dynamic spectrum sensing graph for display on PC 5

6 GPS and Wi-Fi data come only from the Android App 5

7 Android App must run reasonably well on a cellular phone 3

8 Reasonable, competitive cost (FREE) 3

As one can see from the table above, most of the needs center around the Android device/app.

This is due to the fact that all the data-gathering will be done from this device, so it will need to

be efficient and reliable. Although the needs of the customer seem relatively simple, the

implementation is far tougher and will be discussed in detail later.

Product Design Specifications

Due to the fact that this project was largely based in computer programming, we felt that

any potential design specifications were better described as customer needs, which are above, or

as a design implementation, which will be discussed below.

Concept Design

Our advisor, Professor Sidiropoulos, had already done a lot of research on this topic

before handing it over to us to implement. Therefore, many design decisions had already been

made including: using an Android device to take measurements as there is great documentation

to program an Android app and devices were relatively cheap, using Twitter as our data storage

model since that, too, offered a lot of help to be able to implement transference of data from our

Android devices to Matlab, and the use of Matlab as our data processor since Prof. Sidiropoulos

already had extensive knowledge working with Matlab. That being said, many decisions still

needed to be made to successfully reach our goal. Most notable was deciding on how to

interface between the three parts of our overall design. This decision, as well as the choices we

made, will now be discussed at length.

7

Design Description

Android Application

Figure 2: Android App Block Diagram

The data gathering aspect of our project is our Android application. It is designed to

periodically gather the GPS and Wi-Fi information around the Android device that it is installed

on and send that information over Twitter so our PC Twitter Client can further process it. There

are three major components that make up the guts of our application. The Wi-Fi scanner gathers

the MAC addresses and signal strength on each channel of each router. The GPS locator is used

for mapping the Android device’s current location. The Twitter component handles logging into

Twitter and sending off each tweet with our specified hashtag, Wi-Fi, and GPS information. A

textbox was added to be able to send a single tweet for debugging purposes. Additionally a

display at the bottom of the application shows the number of tweets sent and the runtime since

the service has started, the number of GPS and Wi-Fi scans gathered since the last tweet was

8

sent, the Wi-Fi information sent in the last tweet, and the last tweet that was sent formatted to our

specifications.

There is a helper service embedded in our Android application that splits up the Wi-Fi

and GPS components into separate threads. This allows the application to easily run on Android

devices without taking up much of the CPU and memory of the device. It also allows the

application to run each component in parallel and at different frequencies. The Twitter code

comprises of static functions since it needs to be called once to log in and periodically afterwards

once the Wi-Fi and GPS components have finished gathering their information.

The Wi-Fi component runs continuously gathering the signal strength (in dBm) of each

router it is able to find over all eleven available channels for Wi-Fi and categorizes each router

by its MAC address. The reason we categorize each router by their MAC address is that for any

given network each router must have a unique MAC address which allows us to easily

distinguish the different routers. After gathering the Wi-Fi information our application loops

through each Wi-Fi channel and saves the strongest signal strength on that channel with its

corresponding MAC address. This is because the higher the signal strength the more accurate the

measurement is, also there is a limited amount of information each tweet can send and this helps

trim down that information into the most important information to our project.

The GPS component runs periodically set by a timer. This is for many reasons, mainly

Twitter has a cap of 1,000 tweets that can be sent a day which is also broken down into smaller

semi-hourly intervals, but also because the design for our project is to crowed source this

application and there needs to be time for users to move about to gather multiple data points in

different locations. Because of the way that the GPS locator is set up in the Android device when

the function is called to get the devices current location it gathers a set of locations as accurately

as it can, saves those locations and averages them out to get an estimate of the devices location.

It is important to note that our application only uses GPS satellites to get its location and does not

include gathering location information from the routers; this is because our main goal is to

localize the routers and using the routers own information would defeat this purpose.

After the Wi-Fi and GPS information is gathered our application formats the information

into a 139 character string to then be tweeted. The tweet includes our hashtag (#ajd7vF35), the

eleven available channels for Wi-Fi, each channel containing the signal strength compressed

from the two digit dBm to a single character and the last nine characters of the corresponding

9

MAC address, and the latitude and longitudinal coordinates to the sixth decimal point, which if

measured accurately would give to the nearest centimeter.

PC Client

The PC client exists to facilitate communication between the Android app which collects

the data and the Matlab script which analyzes the data. It is implemented with three parts: The

Archivist, a program that searches for Tweets and downloads them, a Java parsing program

which takes in the downloaded Tweets and outputs a Matlab .m file, and a Windows Batch file

that runs it all almost automatically and at start up.

10

Figure 3: PC Client Functional Block Diagram

Detailed Description

The design starts with the Windows Batch file. The Batch file is placed in the startup

folder on the Windows OS so it will run at start up. It then opens the Archivist program and

waits for a user to input in the hashtag and save the results. After the user presses ‘Enter’ it then

runs the Java parsing program which decodes the data and outputs the corresponding Matlab .m

file.

The next part of the design is ‘The Archivist’ program. This program allows a user to

search for Tweets based on a keyword or hashtag. In this case, the user would search for our

hashtag. Once the search finishes, it shows the most recent results (up to 3000 tweets) that have

the keyword. The user can then save these results in a tab delimited text file. The program

automatically updates your search every 5 minutes, so the batch file simply opens it once and lets

it refresh automatically.

The last part of the design is the Java parsing program. This program decodes the Tweets

and stores them in a Matlab format. It starts by asking the user the minimum data points

required for each MAC address. This has to be a minimum of 3 to avoid localization ambiguity

even with perfect distance measurements, but typically it is set to 20 or more to have more robust

data points. Then it searches for the most popular MAC addresses and orders them based on

how many pieces of data each MAC address has. Then, the Tweets are decoded and the program

checks for errors, checks to make sure all data is in the acceptable range, and checks that there

are enough data points to produce accurate results. After this has completed, the program takes

each MAC address that has at least the minimum number of data points and outputs a text file for

each MAC address that contains the Wi-Fi and GPS data in the Matlab format. It also orders

these files from most data points to least. After this the data is ready to be processed in Matlab.

Assumptions and Simplifications

Many things were assumed, such as the basic location of our Tweets, the range of valid

signal strengths, and that the data being received from the Android device was correct.

11

Matlab Script

The final processing of data for this project is handled by Matlab. The goal of the Matlab

script is to take an array of data points as its input and output the location of the best guess

estimate for the Wi-Fi router associated with the input points.

The input array consists of datapoints collected for a single MAC address saved as a .m

file. These input files are run in Matlab to load the variable that contains the data into the

workspace. A global variable which stores the output locations is also initialized. This variable

is an array that has the most recently calculated output location appended to its end.

Once the workspace has been set up, the localization script is callled with a set of datapoints as

its input. These datapoints are stored in a matrix of the form [latitude, longitude, signal

amplitude;]. Using this matrix format takes advantage of Matlab’s ability to perform

mathematical operations on all elements of a matrix. This allows a variable and arbitrary number

of data points to be input to the program. The variable to store the output is also input when the

localization script is called.

The first step to estimating the router location is to convert the received signal strength

from dBm to distance in decimal degrees. The equation governing this relationship is of the

general form:

Where Pr is received power in Watts, Pt is transmitted power in Watts, d is the distance and c

and α are arbitrary constants. Since our power measurments are in dBm, the equation becomes:

or

Where β and α are constants and is received power – transmitted power in dBm. The received

power is measured by the Android app and the transmitted power can be inferred from common

router specifications. β and α must be solved in order to calibrate the equation. This was done

by taking a series of signal strength measurements from known distances with the router in line

of sight to the data collecting phone and then fitting this data until the constants were solved.

The converted distances for each input datapoint are then stored in an array.

The script then uses the Gauss-Newton algorithm to find the point to minimize the mean

square error of the router location guess. This error is equal to the difference of the distance

from each measured point to the estimated point and the measured signal strength converted into

12

distance. The Gauss-Newton algorithm uses successive iterations to converge to a point. Starting

with an initial guess ( ) for the minimum, the method proceeds by the iterations ( )

( ) where ( )

. is the Jacobian matrix of the distance function, which is

equivalent to [(difference of distance from estimated point to measured point and measured

distance on x-axis)/(total error between measured distance and estimated distance), {same but for

y asix}]. This algorithm iterates a set number of times and then outputs the coordinates of the

estimated locations of the WiFi router being localized. These coordinates are added to the array

of calculated output coordinates so that they can easily be plotted all at once.

Design Evaluation

Android App

There have been many iterations of our Android application. To start out we created three

separate applications handling the three major components, Wi-Fi, GPS, and Twitter. To test the

Wi-Fi application we took multiple routers with known MAC addresses and distances from a

testing point and tested to make sure our application could see all of the MAC addresses and

gather appropriate signal strengths of each router. To test the GPS locator we went around and

gathered GPS locations from multiple spots and mapped their location to test the accuracy of the

android devices. One important note to make was that the GPS locator worked wonderfully as

long as there was a good line of site to the GPS satellites. This means that when taking

measurements it is best to be outside, or in a building made out of materials that minimally

interfere with the line of site. This caused trouble when taking measurements inside of the

buildings at the University of Minnesota since most of their buildings are made out of concrete

and rebar which throw off the GPS locator by a wide margin if the locator is able to

communicate with the satellites at all. Testing the Twitter portion of the code was just the

process of logging into a Twitter account, posting a tweet, and verifying on the Twitter website

that the tweet was posted. Finally after the three components were tested we combined them into

a single application with the added helper service to run the components in parallel. The same

tests were then run again to verify that the code continued to work after the merger.

PC Client

13

The Batch file was tested iteratively. A number of frequencies were chosen for how

often it ran, until we settled on every hour.

The Archivist Program was tested by Tweeting specific things from an account and

checking if the program recognized those Tweets and how long it took. We were able to verify

that it was searching for and recording the correct Tweets by just checking our Twitter Accounts

and verifying the information against what The Archivist was outputting.

The Java parsing program was tested by taking in sample tweets first and comparing their

decoding and output to output that was decoded by hand. The results would be decoded by hand

and placed into a Matlab file and the correctness would be verified. After some trial and error,

some minor formatting errors were fixed and the program was verified to be correctly working.

Matlab Script

The initial evaluation testing of the Matlab script was limited to testing the Gauss-

Newton algorithm with dummy data. This data was not generated by the Android app because

the app was not yet completed at the time. The data was created with the distances pre-

calculated in decimal degrees according to their actual distance to the point. This preliminary

testing proved that the algorithm could accurately localize a point if the measured data was

accurate.

When system integration testing was performed it became apparent that GPS

measurements were highly inaccurate when taken inside Keller Hall. It was found that reliable

data could only be gathered under the condition that the phone was outside the building for good

GPS measurements while at the same time having line of sight to a router through a window to

limit the WiFi signal attenuation. After sufficient data was gathered using this methodology it

became possible to calibrate the function for converting signal strength into distance. The script

was then able to be run as the end part of the integrated system of Android app, Windows client

and Matlab. The completed script localized a router location about 2 meters from where the

router actually was using data gathered with proper methodology.

Conclusions and Recommendations

We successfully implemented a solution to our customer’s problem. It contained three

major parts: an Android application that periodically gathers and tweets GPS and Wi-Fi

14

information, a PC Client software that periodically downloads new tweets, decodes them, and

saves them in a format readable by Matlab, and a Matlab script that processes the decoded

information and generates an easy to understand image of the results. Below is a map showing

how our solution estimated the location of an access point in Keller Hall. The red cross shows

the estimated location and the accuracy is within 2-3 meters.

Following are in-depth discussion of how our solution met the customer’s requirements and

potential improvements that can be made to the current solution.

Product Design Specification

Our solution to our customer’s problem successfully met all of the required

specifications. We developed an automated system which was able to successfully localize an

access point based on GPS location and Wi-Fi signal strength measurements made by Android

devices. Specifically, the following customer needs were met:

15

1. The Android application must run in the background; it will continue to gather and report

GPS and Wi-Fi information even if the user navigates to another application or activity.

2. The Android application must not noticeably impact the performance of a typical

Android device.

3. The Android application must encode the GPS and Wi-Fi data in such a way that the

encoded information is able to fit in a single tweet (< 140 characters).

4. The Android application must send tweets at a rate that will provide a good sample of the

spectrum, but does not place undue strain on the device’s battery (~ every 60 seconds).

5. The PC Client software must not noticeably impact the performance of the host computer

it is running on.

6. The PC Client must reject tweets that either were not sent by our application or were not

encoded properly.

7. The Matlab script must generate results in the form of a map or image that can be easily

explained to a casual observer.

There are a number of strengths in our solution to our customer’s problem but also a

number of weaknesses. We will specifically discuss the degree to which this solution is

automated, the implications of only using GPS satellites to gather location data, the user of

Twitter as a medium for reporting data, and the use of Android devices as an information

gathering medium and the concept of “crowdsourcing” this solution for widespread use.

Our goal for this project, both for our own and our customer’s ease of use, was to make

this entire process automated. A current weakness is that we were only partially successful on

that front. The Android application, after it is started, is completely automated; however, the

user must log into a Twitter account and authorize our application to send tweets on his or her

behalf the first time he or she uses the application. We would like to see this part automated as

well. The PC Client software has successfully been automated; it is controlled by a batch script

that can be automatically run by the Windows operating system on the host computer. The

Matlab script is not yet automated and must be manually run on the host computer each time the

user wishes to view new data.

One major weakness of our solution right now is that it only uses GPS satellites to get a

location estimate. This system works well outdoors where there is generally a good line of sight

with the GPS satellites. It begins to break down when the Android devices are indoors and

16

trying to acquire a location estimate. The estimates that the phones get are inaccurate and

difficult to receive at all in many cases. The Android application was written this way with the

localization of access points as the major goal, but in the future when dynamic spectrum sensing

and access are the goals, other resources like Wi-Fi access points and cellular towers can be used

in conjunction with GPS satellites to get a better location estimate inside buildings and crowded

metropolitan areas.

Twitter was chosen as our communication medium for a number of reasons. It has a

well-defined and well-documented API (Application Programming Interface), which means that

the entire infrastructure we’d need to use it was already in place. There is also a lot of sample

code and reference material available which meant that we could quickly begin interfacing with

the product from both Android and Windows. There are, however, some limitations associated

with using Twitter. A Twitter account is only allowed to send so many tweets in a given period

of time, which in turn limits the rate at which we can gather and send GPS and Wi-Fi

information for processing. This also means that we cannot use “bot accounts” that would be the

same across many Android devices because each of those accounts would very quickly reach the

rate limit. Right now, the user must create a separate Twitter account just for this application;

this is not a very user friendly solution. Also, the 140 character limit for a tweet could be a

problem in the future. We successfully fit all the information we are currently gathering into one

tweet, but it is likely that in the future more information will be necessary and 140 characters

will no longer be enough space for it all.

The Android platform was chosen because it is widely used by consumers and it is

extremely well documented. This allowed us to quickly begin developing, testing, and

debugging our application. We were also able to cheaply and easily procure Android devices to

use for live testing of the application. The usage of Android devices also ties well into the idea

of “crowdsourcing” the application for widespread use, because Android devices are among the

most used, with millions of users all over the world.

Ideas for Improvements

Our solution was a first step towards a system for distributed spectrum sensing and

distributed spectrum access. There are many improvements that can be made to improve the

efficiency, robustness, and usability of our solution:

17

Replace Twitter as the communication medium (dedicated servers for receiving and

processing spectrum information)

Extend this application to other radio bands (cellular networks, radio bands, television

broadcast bands, etc)

Improve the user interface for the Android application (allow users to specify how often

information is sent, whether or not application can use their cellular data plan, etc)

Test the Android application on different versions of the Android operation system and

on Android devices from different manufacturers

Completely automate the information processing system (PC Client and Matlab script)

Implement the Matlab script in such a way that it can “learn” from data it has already

processed and automatically calibrate itself on a regular basis in order to return even

more accurate results

“Crowdsource” the application in order to gather data on a much larger scale

Design Process

One of the first things we did when planning our solution to this project was create a

timeline for the development of the solution and a budget. The timeline and budget are attached

at the end of this section.

We ended up using our budget as we had originally planned, to buy Android phones to

use for testing our solution. We did not exactly follow the timeline as we had originally planned

it. We started quickly at the beginning of September, but slowed down in mid-October. We

began to pick up the pace again in the middle of November and were able to finish our solution

on schedule. The slowdown in the middle of October did stop us from doing more testing at the

end of the project, which would have proved useful.

Our design process was generally successful, as we were able to develop all three parts of

our solution in parallel and then bring them together for testing with few integration issues.

Budget

Our budget for this project was $300. We spent that money on three Android phones

used for testing the solution.

18

Schedule

Month: September October November December

Week: 1 2 3 4 1 2 3 4 5 1 2 3 4 1 2 3

Tasks:

Determine initial project scope x x

Develop schedule x x

Draft design specifications x x

Specify program interfacing x x

Order Android phones x x

Design proposal x x

Design review presentation x x

Code Android program x x x x x x

Code Windows client x x x x

Develop MATLAB algorithms x x x x x x

System integration x x x x

System testing x x x x

Product launch presentation x x x

Final report x x x

Milestones:

Put first test app on phone x

Output MATLAB plot x

Import first Twitter message x

19

References

Samsung Galaxy Precedent Specifications:

http://www.samsung.com/us/mobile/cell-phones/SCH-M828CAATFN-

specs

Wi-Fi Specifications:

https://www.Wi-Fi.org/members/certifications-testing/specifications

GPS Specifications:

http://gps.about.com/od/beforeyoubuy/a/howgpsworks.htm

Twitter Development Support:

https://dev.twitter.com/

Matlab Development Support:

http://www.mathworks.com/support/

20

Appendix A: Experience and Resources Available

Personnel:

a. Professor Sidiropoulos, Nikos

b. He, Zhongyu (TA)

c. Corbett, Martin

d. Fiore, Michael

e. Hulke, Alex

f. Malinowski, Grayson

g. Owczarek, Michael

Facilities:

University of Minnesota, Kenneth H. Keller Hall, Lab 3-140.

Equipment/Parts:

a. Lab computer in Kenneth H. Keller Hall

b. 3 – Straight Talk Samsung Galaxy Precedent Android Prepaid Phones

3.2" touch screen display

Android 2.2 operating system

GPS enabled

3G/Wi-Fi connectivity

800MHz processor

Supports microSD cards up to 32GB

Gravity sensor/tilt and shake

Hearing aid compatible (HAC) rating: M4/T4

Battery talk time up to 6 h/Standby time up to 9 days

Includes battery (Lithium Ion), charger, and 2GB microSD card

Model No. STSAM828CPWPSDSV

c. 1 – LG Optimus V Android Phone

3.2" touch screen display

Android 2.2 operating system

GPS enabled

3G/Wi-Fi connectivity

600MHz processor

21

Supports microSD cards up to 32GB

Gravity sensor/tilt and shake

Hearing aid compatible (HAC) rating: M4/T4

Battery talk time up to 5 h/Standby time up to 8 days

Includes battery (Lithium Ion), charger, and 2GB microSD card

Model No. STSAM828CPWPSDSV