Device-to-Device Communication in the Internet of ThingsQSIURP Report

Aliaa Essameldin1 and Khaled Harras2

Abstract— With the quick evolution of the Internet of Things,the research community’s interest is rising in device-to-devicecommunication. In this research, we study the device-to-devicecommunication capabilities of the Intel Edison devices overWi-Fi and Bluetooth and we compare the Edison devices’performance to that of the Raspberry-Pi devices. Further, weexplore potential uses of this communication model by tryingto deploy it on a specific CPS application: Up and Away, awireless UAV cyber-physical test-bed developed in CarnegieMellon Qatar’s Networking Systems Lab.

I. INTRODUCTIONThe Internet we all know, or the network connecting

computers all over the world, is slowly expanding andstarting to include more physical objects, such as sensors andcontrollers. This can potentially allow it to provide smarterservices without requiring the interference of man. This newcomputing paradigm is known as the Internet of Things. TheIoT can serve a wide variety of application domains rangingfrom health and education to green computing and prop-erty management. Fortunately, processors, communicationmodules and most electronic components are diminishing insize and price, which allows us to integrate them into moreobjects and systems. And now that they are being equippedwith network interfaces, the IoT vision has come a lot closerto reality. [4]

With the quick evolution of the Internet of Things, theresearch community’s interest is rising in device-to-devicecommunication. Devices connected to the IoT must be ableto effectively sense, process, communicate and react totheir surroundings [4] in a distributed manner. This taskis specially challenging because IoTs are usually boundby latency or coverage requirements and a limited energysource[6]. Today, there is no doubt about the value of havingdevices within a system seamlessly communicate with eachother independent of human intervention. [1].

The first part of our research involved measuring differentperformance aspects of device-to-device communication. Allinvestigations involved file transfer over a single-hop, but dif-ferent radio technologies (WiFi and Bluetooth) and devices(Intel Edison and Raspberry-Pi). We investigated the impactof two main aspects: communication interface (Bluetooth andWifi) and device (Raspberry-Pi, Intel Edison, Laptop).

Different radio technologies perform differently in termsof coverage and data rates and choosing the ideal technol-ogy depends upon the specific application. In general, for

1Student, Carnegie Mellon University Qatar, Computer Science Depart-ment. essameldin@cmu.edu

2Advisor, Carnegie Mellon University Qatar, Computer Science Depart-ment. kharras@qatar.cmu.edu

Fig. 1. An Intel-Edison Device powered and connected by mini-USB cables

example, some technologies are known to provide higherbandwidth, but higher energy consumption as well, or theopposite. For example, Wi-Fi can provide data rates highenough for transferring high quality videos, but is veryenergy consumptive. On the other hand, 2.4 GHz ZigBeeand sub-GHz technologies provide lower data rates, but muchless energy consumption and are found more appropriate forembedded systems applications that do not require extensivecommunication such as garage doors and light controllers[1].

Since understanding the behavior of different technologiesis so critical in making any IoT system design choices,we investigated the performance of Bluetooth and WiFi interms of delay and coverage. The reason we chose these twointerfaces is that they are the most user friendly. For example,the Intel Edison devices come with Wifi and Bluetoothinterfaces and provides friendly GUIs for controlling themand the Raspberry-Pi devices can support these technologyby simply plugging the convenient USB dongles. Thus, theresults we found will be used by a wide range of IoTdevelopers.

It is also important to investigate device-to-device commu-nication across different embedded devices. The purpose ofsuch investigations is to validate our results by eliminatinga single device’s electric components and kernel implemen-tation as non-experimental factors. It will also contribute asguiding information for making IoT device choice. In thisresearch, we thought it would be sufficient to run our testswith Raspberry-Pi and Intel Edison while using laptops as

control.We have chosen the Intel Edison -?? first because they

were recently released and very widely adopted, so thereis a high, yet unmet, demand for information on their useand performance. Then we chose to validate the results usingRaspberry Pi because it has a large user base so more peoplecan benefit from our research.

The second part of our research involved using device-to-device communication between Edison Intel devices toimprove Up and Away. Such an application is significantbecause advancements in cyber-physical systems have thepotential of improving various industrial areas such asaerospace, transportation, robotics and factory automationsystems. Furthermore, they are predicted to help solve keysocietal challenges such as aging population and economicalchallenges such as scarcity of resources [3].

The usage of CPS in the biomedical or aerospace fields,for example, makes the accuracy and reliability of the systema basic requirement. As a result, the designs become morecomplex and harder to test. The cost of extensive empiricaltesting normally prohibits researchers from carrying out areasonable number of tests on a big enough scale [5]. To meetthis urging need to low-cost and easy-to-deploy experimentsfor CPS the Networking Systems Lab (NSL) is currentlyworking on developing such a Testbed [2].

However, this testbed was centralized. As part of this re-search, we helped the Up and Away team leverage device-to-device communication to upgrade the testbed to a distributedversion of itself. It is a given that a distributed system,if well-designed, has higher scalability and stability than acentralized one.

II. BACKGROUND AND CHALLENGES

Designing the experiments involved making some choiceswith regards to the protocols used either in WiFi andBluetooth. Making such decisions required some in-depthunderstanding of Wifi and Bluetooth stacks. As explainedbelow, making choices with the Wifi tests was straightforward, but we faced a number of challenges when trying torun tests using Bluetooth, which guided our protocol choicesfor the Bluetooth connection.

A. Wifi Set-up

Using the Wifi interface to establish a device-to-device linkwas a fairly simple task. We accomplished this by connectingthe devices in Ad-Hoc mode. An Ad-hoc Network is onein which a set of devices (for our purposes we are onlyusing two) immediately communicate over some wirelesscommunication interface ??. Nodes in an Ad-Hoc networkcan reach other nodes within their radio range without theneed of any routing device. For simplicity, we gave eachdevice a static IP address after it was connected to the Ad-Hoc network. Then, we tested the stability of the connectionusing the pin command.

After thoroughly testing the network layer connection,we had to choose a transport layer protocol to use in theexperiments. The two protocols that best fit our scenario

Fig. 2. A figure showing the Wi-Fi and Bluetooth software stacks

are the Transmission Control Protocol (TCP) and the UserDatagram Protocol (UDP). We ended up using a mix of bothfor the different experiments. Each protocol’s properties anduses in our research are explained in more detail below:

1) Transmission Control Protocol: In a connection-oriented session, a circuit is established through whichpackets flow to the destination. In this arrangement, packetsarrive in order and do not require a full address or otherinformation because the circuit guarantees their delivery tothe proper destination

TCP is a connection-oriented protocol. This means thattwo devices go through a three way handshake that identifiesthe unique connection, or a virtual circuit, between them.This circuit allows TCP to provide some added end-to-end delivery services. For example, this arrangement allowspackets to arrive at their destination without the need of afull address each time and allows the receiver to send anACK for every packet.

2) Datagram Protocol: Connectionless session does notestablish circuits or provide reliable data delivery. Packetsare fully addressed and sent out over the network. TheTransport layer protocols at the destination can re-order thepackets which arrive out of order and request retransmissionof missing or defective packets.

UDP is a connection-less protocol. This means that it sim-ply provides an interface for applications to use the networklayer and send datagrams to identified destinations, but doesnot start a maintainable connection with this destination.UDP provides best-effort delivery; it allows for some dataloss and does not retransmit missing data.

Since UDP provides no extra services, it adds almost noprocessing overhead and is good for media streaming andreal-time gaming. Even though it’s less common than TCP,we had to use UDP in our experiments to measure packetloss and RTT.

B. Bluetooth Set-up

Establishing a Bluetooth connection between two deviceswas a more challenging task. Linux kernels do not recognizethe Bluetooth interface as a network interface. Thus, it cannot

be controlled with the ifconfig command or directly accessedfrom the network libraries like we did with wifi. Instead, weneed a separate interface that provides access to Bluetoothand interfaces to its main protocols. In this project we usedBluez, the official implementation of the Bluetooth stack forLinux.

In the Bluez stack, the Bluetooth radio and base-bandlayers roughly correspond to the physical and network lay-ers in the network stack. On top of those layers lies theL2CAP protocol. L2CAP is responsible for multiplexing databetween higher layer protocols and managing packets. It isoften viewed as the Bluetooth equivalent of UDP.

Bluez provides user-friendly tools, such as the bluetoochctlutility, wwhich allow establishing a connection between twoBluetooth devices and transferring files between them. Usingsuch tools, we made sure that we can connect the devicesproperly and get them to communicate. To run our ownexperiments and tests however, we needed to build applica-tions using Bluetooth. Bluez provides two main approachesto doing this:

1) Bluetooth Programming using RFCOMM: Radio Fre-quency Communication (RFCOMM) emulates the serial ca-ble line settings and status of a serial port and is used forproviding serial data transfer. It maintains a communicationsegment that directly connects the devices on both sides.It provides reliability and error check and, thus, is mostcommonly used for file transfer Bluetooth applications. Ini-tially, we decided to work with RFCOMM since it is themost commonly used bluetooth programming protocol andbecause it has very similar delivery semantics to TCP, whichwould make our wifi/bluetooth comparisons fair.

However, when we started using RFCOMM to transferfiles between Edison devices, we discovered some packetcorruptions that were not handled by RFCOMM. This madeus consider other Bluetooth networking solutions.

2) IP over a BNEP connection: The alternative way ofbuilding an application over a Bluetooth connection is touse the Bluetooth Network Encapsulation Protocol. BNEPis used to transport common networking protocols over theBluetooth media such as IPv4 and IPv6 and is used by thePersonal Area Networking Profile (PAN). The packet formatis based on Ethernet II/DIX Framing as defined by IEEE802.3 (runs directly over L2CAP)

Bluez provides tools for a device to connect to a PANvia BNEP in one of three modes: PAN-User mode (PANU),Network Access point mode (NAP) or Group-Ad Hoc Net-work mode (GN). For simplicity, devices in this project wereconnected in a PANU-NAP connection and given static IPaddresses. Having obtained an IP and a robust Bluetoothconnection over BNEP, we could run the same network TCPand UDP for Wi-Fi and Bluetooth tests. It turned out thatusing IP over BNEP was a better approach since it added tothe consistency of the experiments.

Fig. 3. Percentage of Connections at Nth attempt for each time frame

III. INVESTIGATIONS AND RESULTS

A. Setting-up

We faced no problems when setting the devices to com-municate over a Wi-Fi Ad-Hoc network. When we tried tocreate the RFCOMM/Bluetooth links, however, we faced twomain problems: the connection returns an error if multipleconnections were established in a row and, when it doesconnect and transfer the file, some of the received bytesare corrupted. So we decided to investigate these two issuesfurther.

1) RFCOMM Connectivity: The first observation wasRFCOMM fails to establish multiple connections in a row.So we wrote a client that extensively investigates this issue.When the client fails to connect it gives a Software causedprogram to abort. This was a bug reported in BlueZ 5.28.Users who re-installed blueZ 4.014 were able to avoid it.The bugs means that sometimes the connection betweenpaired bluetooth devices will not start properly. Accordingto Bluez’s documentation, this error can occur when thelocal network system aborts a connection, such as when theSocket interface closes an established connection after dataretransmission fails (receiver never acknowledges data senton a datastream socket).

We suspected that the bug was in Bluez’s implementationof the three-way handshake, it does not properly terminatethe connection. To confirm this hypothesis, we designedan experiment that tests how often connections can besuccessfully established in terms of the time the client waitsbetween closing one connection and starting the other:

• A client connects, then directly disconnects, we dontsend any data.

• It waits for N seconds, then connects and disconnectsagain

• It repeats this a 1000 times• We plotted plot the percentage of successful connections

against the ith connection.The results in -3 confirm that problem is irrelevant from

the duration of the connection and is correlated with thetime waited between connections. We can also observe thatwaiting 0.5 seconds or more ensures that all connections are

Fig. 4. Corrupted Byte during Transmission

Fig. 5. Results of Measuring Transmission Delay against File Size

successful. This shows that the three-way handshake is notimplemented properly (does not block) or connections arenot queued, which are two possible problems with Bluez’ssimplementation of RFCOMM.

2) RFCOMM Correctness: The second observation wasthat some of the bytes during the file transfer were changed.We first wrote a simple application that transfers a file overan RFCOMM socket, and a script on the receiver’s side thatthat compares the received file to a local copy of the samefile. This script detected some byte differences between thetransmitted and original file. We manually checked thoseerrors and indeed there were some replaced bytes. 4 showsa sample from a text file where a lower case n (originaltext, upper paragraph) was replaced with an upper case N(transmitted text, lower paragraph).

So we designed a simple experiment to confirm thepresence of this bug in RFCOMM:

• Both the client and server are connected on the sameAd-Hoc network

• Client starts a TCP connection with the server. It sendsa file of size N in chunks of 1024 bytes, then terminatesthe connection.

• Server received file then runs the file-check script andkeeps counts of correct transmissions.

• This process is repeated 10 times for each file size.• An average of all 10 times is calculates. 6 shows

how time (in milliseconds) varied with file size (inMegaBytes).

• 5 shows the Percentage of successful transmissions foreach file size.

This experiment indeed confirmed the problem. As shown

Fig. 6. Results of Measuring Transmission Delay against File Size

TABLE IHOW DIFFERENT DEVICES BEHAVE WITH RFCOMM

Sender Receiver ResultEdison Edison POSITIVE

Raspberry-Pi Raspberry-Pi NEGATIVELaptop Laptop NEGATIVEEdison Laptop POSITIVE

Raspberry-Pi Laptop NEGATIVE

in the graph, the percentage of correct transmissions de-creases with the increase in file size, which confirms that theincorrect bytes observed are not simple anomalies or imple-mentation errors, but they have a somehow fixed probabilityof occurring.

Having corrupted bytes is a possibility in any radiotransmission process. However, RFCOMM was expected toperform checksum and re-transmit any corrupted packets. Wearrived at two possible scenarios. The first scenario is thatthe corruption happens after the packet passes the RFCOMMlayer and somehow before it’s written to file and, in this case,the bug will be in our application. The second scenario isthat there is a bug in Bluez’s implementation of RFCOMM.

We eliminated the first scenario by writing our own check-sum on top of RFCOMM. Once we did that, we could notlonger see any byte corruptions. To confirm that the problemdisappeared, we transmitted a 20 MegaBytes file 1 100 times,checking its correctness every time at the receiver’s side. Thefile was transmitted correctly at all times.

Then, we tried to narrow the problem down even furtherby trying the same RFCOMM application with a differentcombination of devices. In this set of experiments we startby sending each file-size 10 times and plot percentage againstfile size to confirm that the corruptions fall within the trendobserved earlier. If it does, we identify this experiment asPOSITIVE. In case of no corruptions, we would send the20 MegaBytes file 100 times to confirm the absence of theproblem completely. Once that is confirmed, we identify thisexperiment as NEGATIVE.

As shown in TABLE 1 bytes would only get corrupted ifan Intel-Edison device was involved. This concludes that theissue faced was due to a bug in Bluez’s implementation onYocto (The Linux distribution running on the Inter Edison

Fig. 7. Experimental set-up: Making measurements on different distances

device).

B. Measurements

1) Transmission Delay: The aim of this experiment isto measure how transmission delay over Wi-Fi (TCP/IP)and Bluetooth (TCP/IP over BNEP) varied with file size.Experimental set-up here is simple:

• Both the client and server are connected on the sameAd-Hoc network (in case of Wi-Fi) or PAN (in case ofBluetooth).

• Client starts a TCP connection with the server. It sendsa file of size N 10 times, then terminates the connection.

• Server measures delay in time between first and lastbytes of each sending attempt.

• An average of all 10 times is calculated. 6 showshow time (in milliseconds) varied with file size (inMegaBytes).

It was no surprise that Wi-Fi performed much better thanBluetooth in terms of delay. An interesting outcome of thisexperiment, however, was seeing that the delay was signif-icantly lower transmitting between a laptop and an Edisondevice than between two Edison devices. This was our firstencounter with the Edison’s slow Wi-Fi transmissions. TheRTT measurements (below) reinforced this observation.

2) Round-Trip Time: The aim of this experiment is tomeasure the RTT of single packets and how it increases withdistance. The set-up again is very simple:

• Both the client and server are connected on the sameAd-Hoc network (in case of Wi-Fi) or PAN (in case ofBluetooth).

• Server runs a simple UDP echo server• The client runs a thread that sends the server time-

stamped packets• The client also runs a thread that reads all incoming

echoes and, for each, subtracts the time-stamp on theecho packet from the current time to get the RTT

Fig. 8. RTT Over Distance with Wi-Fi Ad-Hoc

• Client keeps repeating this process until it has 10different RTTs. It then calculates and saves the averageRTT.

• Client moves 5 meters further from the server andexperiment is repeated.

Even though this experiment was initially designed tocompare Bluetooth and Wi-Fi technologies, it served to showus the alarmingly high RTTs resulting from connecting twoEdison devices over Wi-Fi. 8 shows the RTTs againstdistance when we connected the Edison devices and theRTTs against distance when we connected the Raspberry-Pi devices.

The RTTs of the Intel Edison devices are roughly 4 to 5times slower than those of the Raspberry-Pi’s.

C. Application

After we were able to establish seamless device-to-devicecommunication between the Intel Edison and Raspberry-Pidevices, we started helping the UpNAway team develop adistributed version of their CPS test-bed. 9 shows the olddesign of their system. Most of the computation, as shownin the figure, is done in the central node then communicatedto the drones. This team’s ultimate goal is to get rid ofthe central node completely and maintain a fully distributedsystem.

Over the course of this research, we were able to takesome steps in this direction. First, we helped the Up N Awayteam connect an Edison device per drone, the drones andthe central node to a single Ad-Hoc network. The centralnode now runs as a server that maintains the state of eachnode (A node is a drone with its respective Edison device).Then, instead of having all computation done in the centralnode, the central drone now only performs localization andinforms each Edison of its drone’s coordinates. Each Edisoncan then run the drone control modules and pilot the motionof its drone.

IV. CONCLUSIONS AND FUTURE WORKOver the course of this research, we investigated different

areas of working with IoT devices such as the Intel Edison

Fig. 9. Up and Away Old System Architecture [2].

and the Raspberry-Pi devices. We explored how device-to-device communication can be established between suchdevices over either the Wi-Fi or the Bluetooth radio tech-nology. When working with the Intel Edison devices weobserved an ubnormal delay with Wi-Fi packets as well as abug in Bluez’s implementation of RFCOMM. We managedto find an alternative to using RFCOMM, which is usingTCP/IP over a BNEP connection and we tested this solution.However, we did not get a chance to investigate the formerissue with Wi-Fi.

In the future, we should me able to investigate further onwhy the Edison devices have such a long RTT over Wi-Fi,check if it has to do with energy control and try to find a wayround it. Further, we can make more experiments to comparehow these devices perform over different technologies indifferent environments: for example outdoors, indoors inan open corridor or inside cars. Such measurements willcontribute to the design of many developing IoT systems.

REFERENCES

[1] Overcoming challenges of connecting intelligent nodes to the internetof things. 2012.

[2] Amr Mohamed Ahmed Saeed, Azin Neishaboori and Khaled A.Harras.Up and away: A visually-controlled easy-to-deploy wireless uav cyber-physical testbed. 2014.

[3] D. Stovall N. Paine C. Julien C. Fok, A. Petz and S. Vishwanath. Pharos:A testbed for mobile cyber-physical systems. 2011.

[4] Friedemann Mattern and Christian Floerkemeier. From the internet ofcomputers to the internet of things.

[5] L. Sha R. R. Rajkumar, I. Lee and J. Stankovic. Cyber-physical systems:the next computing revolution. 2010.

[6] C. Kaklamanis S. Athanassopoulos, I. Caragiannis and E. Papaioannou.Energy-efficient communication in multi-interface wireless networks.2009.