ShopAssist - A Unified, Interactive, Location-Aware System for Shopping

Bruno LopesInstituto Superior Tecnico

Av. Prof. Dr. Anibal Cavaco Silva2744-016 Porto Salvo, Portugalbruno.lopes@tecnico.ulisboa.pt

Abstract—Being able to locate customers in shopping mallsallows them to appreciate features such as indoor navigation,product localization and enjoy location-based offers. Busi-ness owners may also benefit from it through the producedanalytics, increased revenue and customer engagement. Inthis document we survey the available technologies that makethis possible and present a system which is capable of provid-ing indoor localization services, relying on small Bluetoothdevices – called Beacons. The existing systems are highlyindividual, solely designed for a specific place. Therefore, wedeveloped a generic mobile application which dynamicallylocates users indoors, regardless of the location. Moreover, aContent Management System was also developed to abstractBusiness Owners of the complexity to deploy an indoorlocalization system and to provide an interface for controllingthe media content to be displayed on the Digital Signage.Towards validating our prototype, we performed accuracytests in different scenarios to evaluate the accurateness of thesystem. Likewise, load tests were carried out to assess theperformance of our server, as well as user tests to measurethe usefulness of the system.

I. Introduction

Nowadays, we spend most of our time indoors, eitherat home, at the office or shopping. Over the course oftime, given the broad range of products available, largeretail spaces have essentially turned into the primary ”go-to” place to buy everything. In contrast, small traditionalcommerce is losing its faithful customers. In this typeof proximity commerce, shop assistants usually knowtheir customers and are able to perform a personalizedmarketing by recommending them the products they arelikely to need or buy. The same is true for online com-merce, where massive databases of consumers habits canbe mined for recommendation and targeted promotions,as Amazon does with the recommendation algorithms.However, the same does not apply for large brick andmortar stores, where thousands of individuals cross theentrance doors per day. And in today’s society where timeis at a premium, customers are looking for an efficientshopping, avoiding wasting time searching for goods.

Retail spaces only have access to information aboutthe customer at the time of payment, when he finallyidentifies himself and makes his presence known, justright before he leaves the store! All shopping done bythe customer is considered blind, thus preventing targetedrecommendations to the shopper. Large companies try

to circumvent this by deploying loyalty programs, wherecustomers are regularly contacted (by mail, email or SMS)about global or specifically tailored promotions. Nonethe-less, time spent in the store, browsing products, is a wastedchance for personalized marketing.

Today, there are technologies that may provide a leapforward in this context. As almost every person carries amobile phone, these technologies are available for com-panies to take advantage of.

As we spend most of our time indoors, either workingor shopping, eating, and having in mind that GPS performspoorly inside buildings, a problem emerges. Thus, wecome up with the following question:

How can we identify and locate customers inside stores,in a reliable and non-intrusive fashion?

Currently, there are technologies that aid to overcomethe problem of indoor positioning, and several systemshave already been developed in this scope. However,none of them is completely reliable, as all the underlyingtechnologies have their own drawbacks.

Retail spaces can benefit from indoor positioning tech-nologies that enable them to trigger contextual actionsdesigned by business owners. Pushing unique and highlyindividualized promotions to the customer and collectingdetailed metrics and analytics (heatmaps and dwell timemaps) are some examples that can improve a business’soperation and revenue. Customers can also benefit fromthese technologies to enhance their shopping experience:indoor navigation will facilitate and expedite their visits tolarge stores; augmented reality or contextualized informa-tion display will provide easy access to more informationabout the products that interest them, such as photos,videos and reviews. Some large retail spaces have begunto implement indoor localization technologies and providetheir own mobile apps so that their customers can expe-rience some of these benefits. However, each applicationis designed for a specific context, for a particular place,requiring customers to install a specific app for each store,thereby leading to the following question:

In what way can we overcome the need of oneapplication per place, aiming a deployment in multiple

places?

Taking into account the aforementioned problems andchallenges, the main goal of this thesis is to developa system in which the customer shopping experience isenriched, bringing added value for both sides (businessesand shoppers). Our solution aims to solve the previously

stated problem of individuality, as currently each dif-ferent space requires a different application. To enableindoor navigation, the map of the building is required. Itis intended that new locations can be integrated in ourapplication by supplying the corresponding location databy business owners and desired media content.

II. Indoor Localization

Significant work has already been done in the area ofindoor localization. The technologies, methodologies andsolutions already in use have been surveyed.

A very popular approach for localization is the GlobalPositioning System (GPS). Some attempts to use GPSindoors have been done, namely using high-sensivity GPSreceivers [1] or using Assisted-GPS [2]. GPS performspoorly indoors because buildings block GPS signals, notachieving satisfactory results. Some experiments havebeen using Global System for Mobile Communications(GSM) for indoor localization [3] [4], but none of themcould provide an error margin smaller than 5 meters.Wireless Local Area Network (WLAN) could be a goodchoice due to its widespread deployment, eliminating theneed for installing, extra-hardware, even though it wasnot designed with the purpose of localization. Li et al.[5] stated some problems concerning using WLAN forindoor localization, namely the penetration losses throughwalls, the interference from other sources, and the effectsof signal reflections. Many attempts of using Bluetoothfor indoor positioning took place [6] [8] [7]. The lattertwo used Bluetooth Low-Energy (BLE) – the 4th versionof the Bluetooth Protocol, deployed in 2011 – whicheases and increases localization accuracy by including thedefault RSSI value in the advertisement packet, allowingto achieve an average accuracy of 1.5 meters. Short-range technologies such as Radio-Frequency Identification(RFID) and Infrared (IR) have also been surveyed [9][10], but still not fitting our purpose. Inertial NavigationSystems have the cumulative error problem [11], which isa major drawback. Visual Light Communication Systemssuffer from multipath effects, interferences from otherlamps and require a Line-of-Sight to the light source [12][13].

In conclusion, Bluetooth was the chosen underlyingindoor location technology, as its 4th version mad ein-door localization systems cheaper and easier to procure.Despite the fact radio signal propagation indoors is notregular, it has a reasonable accuracy. Therefore, we chosethe Estimote platform to provide indoor localization ser-vices. This company sells beacons which use BluetoothLow-energy to provide indoor localization, together witha fully documented Software Development Kit (SDK).

There are several methods to estimate a user’s positionindoors that can be used with the above technologies,namely Trilateration and Triangulation, Fingerprinting,Dead-Reckoning and Proximity-based techniques. Someof the analysed technologies can be used with several ofthese methods, while others are only suited to a particularone.

Figure 1. Overall System’s Architecture

III. A Location-Aware System for Shopping

We propose a unified, interactive, location-aware sys-tem for Shopping. The key idea is to take advantage ofthe fact that almost every person carries a mobile phone,by providing customers a context-aware application fortheir devices, while at the same time providing businessowners a way to engage customers and increase revenue.Therefore, our platform is composed by three parts: theServer, the Client and the Digital Signage. We opted for aclient-server architecture due to the need of centralizationof information and because we wanted to provide a uniqueplace for statistic data gathering.

Hence, we developed a modular architecture, in orderto facilitate the extension of the functionalities or their up-dates. Figure 1 presents a detailed view of the architecturein the form of a block diagram of the entire system.

A. ShopAssist Server

The ShopAssist Server is the core of our system. Itis responsible for coordinating the inputs from clients,process them and act accordingly (e.g., by sending con-tent to the Digital Signage display). The Server is alsoresponsible for managing the unique identification of thecustomers, as they can be registered on different stores.It was implemented using the Play Framework and usesa MySQL database to store all locations data. .

Whenever the user device senses a beacon, the Serveris sent the username and BeaconID (Major & Minorvalues of a beacon). This information is processed andmatched to a known location. After this, Location Datais sent to the client, containing all the coordinates ofthe map, beacons position, location of products, windowsand doors. Furthermore, this information from the clientis combined with media content previously stored in theDatabase, and sent to the Digital Signage in order fortargeted advertisement to be displayed.

Our system offers a unified interface where businessowners are able to manage all the information aboutcustomers, locations and media content. The CMS isthe module responsible for this. Therein lies the placewhere business owners can manage or add new locations,administer content to be displayed on the digital signageand consult gathered analytics. It uses a database to storethis information in a persistent manner. The CMS exposesa Web interface with the following functionalities:

• Location Manager - In our platform, a Locationis considered a store. Our Location Manager al-lows the Business Owners to have many storesassociated. It allows the creation of new locationsfrom scratch, or modification of already existingones. In here, business owners can build up themap of his space, add beacons, windows anddoors (to provide better accuracy, as claimed byEstimote).

• Content Manager - Allows assigning media con-tent (images) to each digital signage present onthe corresponding location. Furthermore, it allowsto aggregate customers within user profiles, toassociate advertising campaigns to them.

• Position Log - This feature displays the history ofuser recorded positions of any location belongingto him. It is possible to see on the map whereusers have been, filtering it by time interval. Itis also possible to filter by user. This allows thebusiness owner to analyse the most popular spotsand dead zones of his store, being able to takeactions accordingly, such as the rearrangement ofthe store’s disposition.

The Positioning Engine handles inputs and processesit accordingly, either querying the database, sending thedata to the presentation module of the digital signage or tothe client. It exposes a RESTful Application ProgrammingInterface (API) for the client and the digital signage to use.

B. Client

The client runs our mobile application on his smart-phone, which provides indoor positioning and allows cus-tomers to browse for products within the current loca-tion. It displays a list of products or product sections(whichever the Business Owner defines) in the mainscreen. The selected product is then marked in the map,enabling customers to guide themselves to the destination.The application also notifies the presence of the user inthe store, causing the Digital Signage to be activated.

The client acquires Bluetooth data (RSSI and Bea-conID) through the Bluetooth Module and passes it tothe location module. The BeaconID consists of a pairof numbers (Major and Minor) which uniquely identifyeach beacon within a location. The Location Module isa wrapper around the Indoor Positioning System fromEstimote, which provides positioning within a space. ThePresentation Module displays a map of the location wherethe user is located.

Regarding the interactions, the Client (mobile applica-tion) queries the Server through a REST API to performthe user authentication. In case of a successful authenti-cation, the application starts a loop listening for beacons,

Figure 2. ShopAssist iOS mobile app screenshot

displaying a message on the mobile screen saying ’Waitingfor a beacon’. Once a beacon is sensed, the loop ends andthe beacon’s major and minor ID along with the loggedusername are sent to the server. Then, the server return aJSON message containing all the location data (its points,windows, doors, and so on). This data is used setup thelocation object for further indoor positioning in the mobileapplication.

Then, the positioning loop begins, where the user’sposition is calculated by Estimote framework. These co-ordinates are used to draw the avatar on the map, whichmoves according to users movement. Subsequently, theapplication kicks off a background task to send a POSTrequest to the Server to store the user’s position on thedatabase.

In case the user is closer by less than one meter to aDigital Signage, the beacon’s major and minor ID, alongwith the username are sent to the server. The server thenacts accordingly, querying the database in order to retrievecontextual data (media content) for that user. Finally, itupdates the Digital Signage with that content so it can bedisplayed to the user.

C. Digital Signage

Consists of having several screens spread around theshopping mall to display targeted advertisements. Uponapproach of an user to the screen, it displays potentiallyattractive promotions based on his user profile.

The Presentation Module of the Digital Signage isresponsible for displaying media content, sent by theShopAssist Server. It consists of a screen which showsindividualized promotions for the customer nearby.

There is a REST API that is called to indicate thepresence of the customer in the proximity and trigger

actions accordingly. Moreover, the campaigns and mediacontent are retrieved from the database, registering howmany times a user has seen them.

Each Digital Signage has a Raspberry (A small-sizedcomputer) with a webpage opened in fullscreen. This pagedisplays a continuous loop of default images (chosen bythe Shopkeeper), and upon user approach it silently startsdisplaying targeted images for the nearby user.

Every digital signage has a beacon associated with it,normally positioned above, as seen on picture 3.

Figure 3. Digital Signage running with a Raspberry Pi 3, with a beaconabove.

To keep the content of the digital signage updated, weneed an alive connection with the server.

With HTTP request-response method, the digital sig-nage module would have to periodically make GET re-quests to the server to know if there is new content tobe displayed. The server could not send data unless thedigital signage explicitly requested it.

Hence, a Websocket connection between the digitalsignage and the server is maintained, allowing the serverto immediately send updated data once available.

IV. Evaluation

We performed accuracy tests on different places, withdifferent devices, to observe how the accurateness of theindoor positioning varied. We also run load tests on ourserver to assess its response time and resources consump-tion at unusually high loads. Likewise, we measured theresources consumption of our mobile application. More-over, user tests were carried out to survey the usabilityof our prototype as a whole and to collect the suggestedimprovements.

A. Accuracy Tests

Accuracy tests were made in two different locations(an university office and a room of an apartment), withtwo different devices: an Apple iPad Mini 2 and iPhone6s).

Figure 4 presents an histogram of the error distri-bution, measured with both used devices (iPad Mini 2and iPhone 6s), on an university office. We can observethat the iPad achieved a lower error margin. We wereexpecting to achieve a better accuracy from the iPhone

6s, as Estimote Indoor Location claims to be better tunedaround iPhone’s rather than iPads. But surprisingly, theiPhone reported a lower accuracy comparing to the iPad,under the same test scenario.

0

2

4

6

8

10

12

14

≤ 0.50 ]0.50; 1.00] ]1.00; 1.50] ]1.50; 2.00]

Coun

t

Error [m]

Frequency Distribution in Office

iPad Mini 2 iPhone 6s

Figure 4. Result of the accuracy tests in Office

From the 32 taken measurements of the accuracy testsconducted in a room of an apartment, we estimated aprobability interval for 95 percent confidence using a t-student distribution. By the made calculations, we can be95% confident that our accuracy error will fall between0.942 and 1.272 meters.

Overall, the accuracy tests carried out resulted on anaverage error of 1 meter, which we find acceptable foran emerging technology in the field of indoor localizationthe Bluetooth Low-Energy. If the average error remainsthe same in bigger locations, such as large retail spaces,the impact will be smaller. Nevertheless, it does not offerenough accuracy to find specific products within storeshelves.

B. Load Tests

We tested our server under an increasing number ofclient requests and measured the network, memory andCPU usage of the server. We made six different tests,doubling the amount of requests. We queried the serverwith 1, 10, 20, 40, 80 and 160 requests per second, duringone minute. In other words, these numbers mean clientrequests made each second. For instance, it lets us knowhow does our server perform when 160 clients connectevery second, over a 1 minute period (totalling 9600requests on one minute).

From the load tests performed, we obtained a responsetime of less than 6 seconds when querying the server with160 requests per second, over a minute. As having 160users starting the ShopAssist Mobile application in theexact same is a scenario that requires the adoption of oursystem by a large number of users, we can conclude thatour server responds in a acceptable time under heavy-load.

We also measured the network, memory and CPU us-age of the server. Regarding memory usage, we observedthat memory usage of both Java and MySQL processesbarely fluctuated, allowing us to confirm that our Javacode of the Server does not have any memory leak. TheCPU usage of the server was also not significant, with alarge margin for expansion: 40% usage of one core out of8 total cores, under 160 requests/second. Regarding the

network usage, we measured a transfer rate lower than30 kB/s under 160 requests/second, confirming that usingJSON for data exchange was a good choice.

C. User Tests

From the user tests conducted, we observed the mainfaced difficulties allowing us to understand where thesystem can be improved, concerning user experience.We encountered significant differences in the time-of-accomplishment of the tasks among users, which ex-plained by the fact that some participants felt the applica-tion easier to operate than others.

The participants who used the system filled a question-naire with 7 statements. The questionnaire had a 7-pointLiker-scale, where 1 corresponded to Strongly agree, and7 corresponded to Strongly disagree. The users classifiedpositively our application in terms of simplicity and easeof use, but assigning the worst classification to statement 3”I am satisfied with system accuracy.”. This classificationcould be explained by the too high expectations users had,by the fact that users were not holding the iPad in flatposition, and because of the small test scenario, where anerror of 1.5 m is half the width of the office a significantamount.

D. Mobile App Resource Usage

We measured the resources consumption of our mobileapplication by monitoring its CPU, RAM, Network andenergy usage.

Regarding the CPU usage, an average 6% of CPUusage is obtained after some time running the application.This percentage is explained by the complex algorithmsbeing executed in the background while Estimote Indoorframework is calculating user position. Also, some of thisconsumption might be explained by our position logger,which reports the user position to the server every second.

Concerning Network usage, the application uses rel-atively low bandwidth. After receiving the initial JSONmessage from the server, only a minimal upload band-width is used to submit periodically the current userposition to the server

We observed that our app has an high energy demand.This is mainly due to the CPU usage, as our app doescomplex calculus in order to determine a position, whichdemands processment and energy. Also, using the networkoften to report the position to the server consumes energy.Moreover, despite the fact that we are using BluetoothLow-Energy, it still requires energy to operate.

E. Cost estimation

To estimate the costs of deployment in a larger scale,we assume a small-sized store. The average size for asmall discount store is between 200 - 800 square meters.Hence, we consider an area of approximately 230 sq.meters (18.50 * 12.50 m) as an acceptable value for ourcost estimation. Regarding the ideal number of beacons,it varies according to the user experience the businessowner wants to create. ie., the higher the desired numberof digital signage, the greater the number of necessarybeacons.

Name Price Qty Total

Estimote Dev Kit (3 beacons) 100e 5 500e

Raspberry Pi 3 45e 2 90e

Samsung S24C450B Monitor 195e 2 390e

Total cost: 980eTABLE 1. COST ESTIMATION FOR AN APPROXIMATELY 230 SQ.

METER STORE

As observable in 1, our cost estimation for a small-sized store is of 980e. We find this value accessible tomany stores, motivating us to continue the path of thiswork.

V. Conclusion and Future Work

The goal of this thesis was to develop a system toidentify customers and help them locate products in stores,enriching users shopping experience and bringing addedvalue to the business owners. To achieve this, Bluetoothwas the chosen underlying indoor location technology.Additionally, we chose the Estimote platform to provideindoor localization services. This company sells beaconswhich use Bluetooth Low-energy to provide indoor local-ization, together with a fully documented SDK.

Our system presents an interface for business own-ers – the Content Management System – that abstractsbusiness owners from the complexity of deploying a bea-con infrastructure to provide indoor localization services.Additionally, our prototype was also designed in tandemwith the Digital Signage and Mobile platforms. The firstprovides users a context-aware shopping experience, withtargeted content being shown in the Digital Signage. Thelatter adds an interactive layer over traditional shopping,allowing users to locate themselves within the store andbrowse for products in the mobile application.

Presently, each different retail space requires a spe-cific application. Our main goal was to deploy a genericsystem where Business Owners solely need to providethe corresponding map information along with the desiredmedia content to be displayed in Digital Signage. Thisway we save the customer from the unpleasantness ofinstalling 10 different apps for 10 different stores. Thecustomer only needs to install one mobile application,which works in multiple stores, and thus increasing thechances of adoption of the app.

In other words, we present a layer over EstimoteFramework facilitating the deployment of a context-awaresystem, mixing advertisements with usefulness (indoorpositioning, locating products, targeted advertisements).The opportunities are endless, and some that are worth tomention were left for Future Work due to time constraints.

Overall, we accomplished a good accurateness fromour prototype, with error distances hovering around 1meter.

Some features were left for future work due to timerestrictions, which we would like to enhance in the future.We would like to extend out test scenario into a morecomplex space, ideally a retail shop, in which we couldtest the real application of our platform.

Some possible ideas for further improvement areadding support for video media content, as an animated

and more vivid Digital Signage would attract the attentionof more customers. Besides this, another enhancement toour system can be incorporating map-caching. Allowingan offline navigation mode for previously visited storeswould dispense the need for requesting the map every timethe customer visits the same store. Furthermore, althougha unique and generic mobile application is advantageousfor customers, in order to increase the chances of adoptionby business owners, we would have to provide morepersonalization and flexibility, i.e., displaying promotionsin the mobile phone, in-app purchases, and so on.

References

[1] Kjærgaard, M.B., Blunck, H., Godsk, T., Toftkjær, T., Christensen,D.L., Grønbæk, K.: Indoor positioning using gps revisited. InFloreen, P., Kruger, A., Spasojevic, M., eds.: Pervasive Computing.Volume 6030 of Lecture Notes in Computer Science. Springer BerlinHeidelberg (2010) 38–56

[2] Djuknic, G., Richton, R.: Geolocation and assisted GPS. Computer34(3) (March 2001) 123–125

[3] Caffery, J., Stuber, G.: Overview of radiolocation in CDMA cellularsystems. Communications Magazine, IEEE 36(4) (April 1998) 38–45

[4] Otsason, V., Varshavsky, A., LaMarca, A., de Lara, E.: Accurate gsmindoor localization. In Beigl, M., Intille, S., Rekimoto, J., Tokuda,H., eds.: UbiComp 2005: Ubiquitous Computing. Volume 3660 ofLecture Notes in Computer Science. Springer (2005) 141–158

[5] Li, B., Salter, J., Dempster, A.G., Rizos, C.: Indoor positioningtechniques based on wireless LAN. In: First IEEE InternationalConference on Wireless Broadband and Ultra Wideband Communi-cations, Citeseer (2006)

[6] Weissman, Z.: Indoor location. White paper, Tadlys Ltd (2004)

[7] Taylor, N.: Marketing with Mobile Push Notifications in a LocationSpecific Context. Technical Library (2014)

[8] Corbacho Salas, A.: Indoor Positioning System based on BluetoothLow Energy. Master’s thesis, Universitat Politecnica de Catalunya(June 2014)

[9] Ni, L.M., Liu, Y., Lau, Y.C., Patil, A.P.: LANDMARC: IndoorLocation Sensing Using Active RFID. Wireless Networks 10(6)(November 2004) 701–710

[10] Want, R., Hopper, A., Falcao, V., Gibbons, J.: The active badgelocation system. ACM Transactions on Information Systems 10(1)(January 1992) 91–102

[11] Evennou, F., Marx, F.: Advanced Integration of WiFi and InertialNavigation Systems for Indoor Mobile Positioning. EURASIPJournal on Advances in Signal Processing 2006 (January 2006) 1–12

[12] Liu, X., Makino, H., Maeda, Y.: Basic study on indoor locationestimation using Visible Light Communication platform. Conferenceproceedings : ... Annual International Conference of the IEEEEngineering in Medicine and Biology Society. IEEE Engineering inMedicine and Biology Society. Annual Conference 2008 (January2008) 2377–80

[13] Yang, Z., Wang, Z., Zhang, J., Huang, C., Zhang, Q.: Wearablescan afford: Light-weight indoor positioning with visible light. In:Proceedings of the 13th Annual International Conference on MobileSystems, Applications, and Services, ACM (2015) 317–330