1

Learning-based Computation Offloading for IoTDevices with Energy Harvesting

Minghui Min, Student Member, IEEE, Liang Xiao, Senior Member, IEEE, Ye Chen,Peng Cheng, Member, IEEE, Di Wu, Senior Member, IEEE and Weihua Zhuang, Fellow, IEEE

Abstract—Internet of Things (IoT) devices can apply mobileedge computing (MEC) and energy harvesting (EH) to providehigh level experiences for computational intensive applicationsand concurrently to prolong the lifetime of the battery. In thispaper, we propose a reinforcement learning (RL) based offloadingscheme for an IoT device with EH to select the edge deviceand the offloading rate according to the current battery level,the previous radio transmission rate to each edge device andthe predicted amount of the harvested energy. This schemeenables the IoT device to optimize the offloading policy withoutknowledge of the MEC model, the energy consumption model andthe computation latency model. Further, we present a deep RLbased offloading scheme to further accelerate the learning speed.Their performance bounds in terms of the energy consumption,computation delay and utility are provided and verified viasimulations for an IoT device that uses wireless power transferfor energy harvesting. Simulation results show that the proposedRL based offloading scheme reduces the energy consumption,computation delay and task drop rate and thus increases theutility of the IoT device in the dynamic MEC in comparisonwith the benchmark offloading schemes.

Index Terms—Mobile edge computing, energy harvesting, re-inforcement learning, Internet of Things, computation offloading.

I. INTRODUCTION

With limited computation, energy and memory resources,Internet of Things (IoT) devices such as sensors, camerasand wearable devices have a computation bottleneck to limitthe support for advanced applications such as interactiveonline gaming and facial recognition applications [1], [2]. Thischallenge can be addressed by mobile edge computing (MEC)techniques [3], in which IoT devices offload the computationtasks to edge devices such as the base stations (BSs), access

The work of L. Xiao was supported in part by National Natural ScienceFoundation of China under Grant 61671396 and 91638204, and in partby the open research fund of National Mobile Communications ResearchLaboratory, Southeast University (No.2018D08). The work of D. Wu wassupported by the National Natural Science Foundation of China under Grant61572538, the Fundamental Research Funds for the Central Universitiesunder Grant 17LGJC23, Guangdong Special Support Program under Grant2017TX04X148. (Corresponding author: Liang Xiao).

Minghui Min, Liang Xiao and Ye Chen are with the Department ofCommunication Engineering, Xiamen University, Xiamen 361005, China.

Peng Cheng is with the State Key Laboratory of Industrial Con-trol Technology, Zhejiang University, Hangzhou 310027, China (email:pcheng@iipc.zju.edu.cn).

Di Wu is with the School of Data and Computer Science, Sun Yat-Sen University, Guangzhou 510006, China, and also with Guangdong KeyLaboratory of Big Data Analysis and Processing, Guangzhou, 510006, China(email: wudi27@mail.sysu.edu.cn).

Weihua Zhuang is with the Department Electrical and Computer Engi-neering, University of Waterloo, Waterloo, ON N2L 3G1, Canada (e-mail:wzhuang@uwaterloo.ca).

points (APs), laptops and even smartphones within the radioaccess of the IoT devices. By utilizing the computation, cacheand energy resources of edge devices, computation offloadingcan reduce the computation delay, conserve battery resources,and even enhance security for the computation intensive IoTapplications [4], [5].

Energy harvesting (EH) is another promising technique thatprolongs battery lifetime and provides a satisfactory quality ofexperiences for IoT devices [6]–[9]. The EH module enablesan IoT device to capture the ambient renewable energy such assolar radiation, wind power generation, radio-frequency (RF)signals, and kinetic human motion to supply the “greener”energy for the IoT central processing unit (CPU) and the radiotransceiver.

One critical problem in the computation offloading is theselection of the edge device from all the potential radio devicecandidates within the radio coverage area [10]. This is furthercomplicated by the determination of offloading rate, i.e., thequantity of computation tasks to offload to the edge device[11], [12]. An IoT device often requires a longer period of timeto transmit the offloading data and receive the computationresult compared with the local computation within the IoTdevice. This is further complicated by the chosen edge devicecarrying out heavy workloads and experiencing degraded radiochannel fading and interference [9]. It is therefore challengingfor an IoT device to optimize the offloading policy within thedynamic MEC network with time variant radio link transmis-sion rates [13], especially with an unknown amount of therenewable energy within a given time duration.

The complications associated with offloading have recentlyattracted the attention of researchers. For example, the mobileoffloading scheme as proposed in [14] uses the Lyapunovoptimization to minimize the worst-case expected computationcost with a single known MEC server, based on the knowledgeof both the transmission delay model and the local executionmodel. The online learning based MEC in [15] which utilizesEH can reduce the time cost, assuming both the normaldistributed renewable energy generation under the known of-floading delay model and the power consumption model. Theassumptions may not be practical for IoT devices, especiallyin dynamic MEC networks with multiple edge candidates.

In this paper, we investigate computation offloading of IoTdevices with EH in a dynamic MEC network. We considerthat an IoT device is connected with multiple edge devicesof different communication overheads, and can utilize the EHhistory or model to approximately predict the future renewableenergy generation trend. We present a reinforcement learning

2

(RL) based computation offloading framework for an IoTdevice to select an edge device and consequently determine theproportion of the computation tasks to offload. Each offloadingdecision is made in accordance to the radio transmission rateof each IoT-edge link in the previous time slot, the predictedrenewable energy harvesting model and the current batterylevel of the IoT device. The computation offloading process ofthe IoT device can be modeled by a Markov decision process(MDP). Consequently, reinforcement learning techniques, suchas Q-learning, enable the IoT device to act as a learning agentto achieve the optimal offloading policy after a large numberof offloading interactions with probability one [16].

We propose a reinforcement learning based computationoffloading scheme (RLO) for an IoT device with EH withoutknowledge of the MEC model, the computation delay model,and the energy consumption model. The offloading policyincluding the edge device and the offloading rate is selectedaccording to the current MEC state and the quality function orQ-function that is the expected long-term discount reward foreach state-action pair. The IoT device evaluates the reward orutility value based on the overall delay, energy consumption,task drop loss, and the data sharing gains within each time slot.This in turn updates the Q-function according to the iterativeBellman equation [16] based on the utility and the latestoffloading policy. A transfer learning method in reinforcementlearning [17] exploits the offloading experiences in similarscenarios to initialize the Q-values and therefore reducing theexploration time at the initial stage in the repeated offloadingprocess with edge devices.

Moreover, we propose a deep reinforcement learning basedoffloading scheme (DRLO) to improve the offloading perfor-mance for the case with a large state space. For example,an IoT device can expect a large number of feasible batterylevels, a large amount of renewable energy generated in a timeslot, and a large number of potential radio transmission ratescorresponding to each edge device. This scheme applies deepconvolutional neural network (CNN) [18] to compress the statespace and accelerate the learning speed. This scheme can beimplemented on IoT platforms such as Qualcomm Snapdragon800 and Nvidia Tegra K1 that support for deep learning [19],[20].

We present the performance of RL-based offloadingschemes and the convergence performance for three typicalMEC scenarios, for an energy consumption model based onCPU frequencies and transmission power. The local compu-tational time and the transmission time are considered in theanalysis, and the performance bound of the proposed schemeregarding the energy consumption, computation delay, andutility from each offloading decision is provided. Simulationsare performed for IoT devices with wireless power transfer tocapture the ambient RF signals from a dedicated RF energytransmitter to charge the IoT battery [21], [22]. Simulationresults show that the DRLO scheme saves the energy con-sumption of IoT devices, reduces the computation delay anddecreases the task drop rate, and thus increases the utilityof IoT devices, compared to the benchmark Q-learning-basedoffloading scheme. The main contributions of this study aresummarized as follows:

(1) An RL-based computation offloading scheme for IoTdevices with EH is presented. This scheme enables an IoTdevice to select the edge device and the offloading rateaccording to the current battery level, the previous radio trans-mission rate, and the predicted EH model. Further, this schemealso enables an IoT device to achieve the optimal offloadingwithout knowing the MEC model, the energy consumptionmodel and computation delay model in dynamic MEC withlow computation complexity;

(2) We propose a deep RL-based offloading scheme tocompress the state space dimension, with a transfer learningtechnique to accelerate the learning speed at the initial stagesand improve the offloading performance for IoT devices withsufficient computation resources;

(3) We provide the performance bound of the proposed RL-based offloading schemes and prove their convergence to theoptimal utility.

The rest of this paper is organized as follows: Relatedstudies are reviewed in Section II. The mobile offloadingmodel and the EH model are presented in Section III. AnRLO scheme for IoT devices with EH is proposed in SectionIV and a DRLO scheme is discussed in Section V. Analysisof the performance bounds of the proposed computationoffloading schemes is given in Section VI. Simulation resultsare discussed in Section VII, followed by the conclusion ofthis study in Section VIII.

II. RELATED WORK

MEC is an effective approach to augment computationcapability of mobile devices. An offloading framework isapplied for offloading the computation tasks from a singlemobile device to multiple edge devices, and the task alloca-tion and CPU frequency are jointly optimized to minimizethe execution latency and energy consumption [10]. Binaryoffloading proposed in [23] reduces the computation overheadfor resource-constrained mobile devices with multi-channelwireless interference with a single edge. Resource allocationand partial offloading in a multiuser MEC network are inves-tigated to minimize the energy consumption under a latencyconstraint [24]. Dynamic voltage scaling is incorporated intopartial computation offloading to jointly decrease the energyconsumption and latency in the allocation of the computationresources, transmission power, and offloading rates of themobile devices [25].

Energy harvesting is a promising solution to provide per-petual energy supply for self-sustainable mobile devices. ALyapunov optimization-based binary offloading scheme isproposed for the EH-powered mobile devices to reduce theexecution delay and computation failures with a single knownMEC server [14]. For EH-powered edge servers, a learning-based online algorithm is studied for the system operatorto decide the amount of workload offloaded from the edgeserver to the central cloud and the processing speed of theedge server [15]. The study is based on the information ofcore network congestion state, computation workload, andenergy state information. A wireless-powered single-user mo-bile cloud computing system is investigated in [11], which

3

includes the evaluation of the optimal CPU cycle frequenciesfor local computing and time division for offloading to adaptto the transferred power, based on the convex optimizationtheory. Furthermore, a binary offloading policy in a wireless-powered MEC network is proposed, which utilizes the APto transfer RF power to the mobile device and execute thecomputation tasks [12]. Most of existence works are basedon the known models, such as transmission delay model andenergy consumption model. However, these assumptions maynot be practical for IoT devices in dynamic MEC networks.

RL techniques have been applied for offloading strategiesin the dynamic MEC network. An efficient RL-based resourcemanagement algorithm is proposed to optimize the on-the-flyworkload offloading rate to centralized cloud and edge server,provisioning to minimize the service delay, and operationalcost [15]. The computation offloading strategies include ap-plication of Q-learning techniques, i) to derive the optimaloffloading rates and reduce the attack rate of smart attackers atIoT devices [26], and ii) to achieve optimal offloading and re-duce response time in edge-based offloading [27]. Q-learning,Dyna-Q and post decision state techniques are applied tomalware detection offloading in order to improve the detectionperformance without being aware of the trace generation andthe channel models in dynamic radio environments [28]. Abinary MEC offloading scheme named DRL in [29] uses deepRL to choose whether to offload to the edge device thatserves multiple users. This can reduce the energy consumptionand average computation delay as compared with both thefull-offloading and non-offloading schemes. This work furtherimproves the offloading scheme by incorporating the radiotransmission rate, the harvested energy and the battery level inthe state of the learning process and using transfer learning anddeep learning to optimize the edge selection and the offloadingrate for better computation performance.

III. SYSTEM MODEL

An MEC network consisting of an IoT device and M edgedevices is considered, as shown in Fig. 1. The IoT devicecan represent a smart watch and smartphone [30], which hasto support computation intensive tasks such as augmentedreality applications. These tasks should be processed timelyto satisfy the user experience and to guide further actionslocally or remotely. The IoT device is equipped with electricitystorage elements and EH components, such as RF energy har-vester, photovoltaic modules, and wind turbines. With energypowered by the EH component, the IoT device can eitherexecute its computation tasks locally or offload partial or alltasks to the edge devices that can provide high computationperformance with a virtual machine. Time is slotted with aconstant slot duration, and the time slot index is denoted byk, with k ∈ {1, 2, ...}.

The IoT device generates computation tasks associated witha data volume of C(k) bits at time slot k. According toliterature [31], we focus on delay-sensitive applications, withexecution duration of the computation tasks C(k) no longerthan a time slot length. The IoT device applies the computationpartition scheme proposed in [32] to divide the computationtasks into Nx parts.

xkCk

bk

Bik

M

k

i

xkCk

Fig. 1. MEC system with EH-powered IoT devices.

TABLE ILIST OF NOTATIONS

Symbol Description

x(k)Proportion of the data offloaded to the edgedevice at time slot k

B(k)i

Radio transmission rate between the IoTdevice and edge device i

ρ(k) Amount of harvested energyb(k) Battery levelP Transmit power of the IoT deviceη Transmit power of the energy transmitter

fScheduled CPU-cycle frequencies for localexecution

NNumber of CPU cycles required to processone bit task input

E(k) Energy consumptionT (k) Computation delayψ Task drop lossW Experience size in the CNN input sequenceH Size of minibatch of the CNN

The IoT device selects edge device i to offload the compu-tation tasks at time slot k based on radio link transmission rateB

(k)i , with 1 ≤ i ≤M , and decides the offloading rate denoted

by x(k), with x(k) ∈ {l/Nx}0≤l≤Nx . More specifically, the IoTdevice executes the computation tasks locally if x(k) = 0, andoffloads all the computation tasks to edge device i if x(k) = 1.The IoT device offloads x(k)C(k) data to the selected edgedevice and the remainning (1− x(k))C(k) data are processedlocally if 0 < x(k) < 1. The IoT device selects its offloadingpolicy at time slot k denoted by a(k) = [i(k), x(k)] from allthe possible policies denoted by A. For ease of reference, ourcommonly used notations are summarized in Table I. The timeindex k in the superscript is omitted if no confusion occurs.

A. Local Computing

The CPU of the IoT device is the primary engine forlocal computation. The performance of CPU is controlledusing the CPU-cycle frequency. Local computing for executing(1−x(k))C(k) input bits occurs at the IoT device. Let N denote

4

the number of CPU cycles required for computing one inputbit. Therefore, the total number of CPU cycles required for the(1−x(k))C(k) bits is (1−x(k))C(k)N . By applying dynamicvoltage and frequency scaling techniques [25], the IoT devicecan control the energy consumption for local task executionby adjusting the CPU frequency fm for each cycle m, withm ∈ {1, 2, ..., (1 − x(k))C(k)N}. The execution latency forlocal computing at time slot k is denoted by T

(k)0 and given

by the following equation:

T(k)0 =

(1−x(k))C(k)N∑m=1

1

fm. (1)

According to [23], the IoT device consumes E(k)0 energy

for local computing at time slot k, with

E(k)0 =

(1−x(k))C(k)N∑m=1

ςf2m, (2)

where ς is the effective capacitance coefficient that dependson the chip architecture. The proposed scheme is applicable toIoT devices with the dynamic frequency and voltage scalingaccording to [23] and [25], and fixed CPU frequency andvoltage can be viewed as a special case.

B. Computation Offloading

The IoT device sends the computation tasks to the edgedevice i at time slot k with the uplink radio transmission rateB

(k)i . The transmission duration to offload x(k)C(k) bits data

to edge device i is represented by T (k)i , with

T(k)i =

x(k)C(k)

B(k)i

. (3)

Based on [33], the IoT device consumes E(k)i energy to offload

the tasks to edge device i at time slot k, which depends on thetransmit power P for offloading and the transmission durationT

(k)i , with

E(k)i =

x(k)C(k)P

B(k)i

. (4)

C. Energy harvesting

The IoT device may contain RF energy harvester, photo-voltaic modules and wind turbines, which can convert renew-able resources (such as ambient RF signals, wind and solar)to electricity. It also contains a battery in order to balance thepower supply and demand. Moreover, the harvested renewableenergy can be stored in the battery to support both the localcomputing and computation offloading.

The amount of energy harvested at time slot k is denotedby ρ(k) and the total energy consumed by the IoT device attime slot k is denoted by E(k), with E(k) =

(E

(k)0 + E

(k)i

).

The battery level at the beginning of time slot k is denoted byb(k), and it evolves according to the following equation:

b(k+1) = max{0, b(k) − E(k) + ρ(k)

}. (5)

The IoT device drops the computation task, if its energy isinsufficient, i.e., b(k+1) = 0 [14].

As a special case, the RF-enabled wireless energy transfertechnology in [21] is considered, in which the IoT device ispowered with the dedicated wireless power transmitter thatprovides continuous and stable microwave energy over the air.Let υ ∈ (0, 1) be the energy conversion efficiency, η(k) denotethe transmission power, d(k) represent the distance betweenthe energy transmitter and the IoT device at time slot k,τ ≥ 2 denote the path loss factor exponent, and G denote thecombined gain of the RF energy transmitter antenna and theIoT antenna. According to [21], the amount of the harvestedenergy of the IoT device in time slot k is given by

ρ(k) = υη(k)(d(k)

)−τ

G. (6)

The amount of energy harvested at time slot k can beestimated by the IoT device according to the power harvestedhistory and the modeling method in [34]. The estimatedamount of the generated energy is denoted by ρ̂(k). Theestimation error regarding ρ(k) is denoted by △(k), whichfollows a uniform distribution with the maximum estimationerror ζ, and is given by

△(k) = ρ(k) − ρ̂(k) ∼ ζU(−1, 1). (7)

IV. RL-BASED COMPUTATION OFFLOADING

In the dynamic computation offloading process, the IoTdevice selects its proportion of data to offload to an edgedevice based on the system state, consisting of the previousradio link transmission rate, the predicted amount of harvestedrenewable energy, and the current battery level. Noteworthy,the next system state observed by the IoT device depends onlyon the current state and actions, and not on the past historyof the process. Therefore, the computation offloading processcan be viewed as an MDP, in which the Q-learning technique,a model-free and widely used RL technique, can derive theoptimal policy without knowledge of the MEC model, energyconsumption model, and computation latency model.

We propose an RL-based offloading scheme RLO that usesthe transfer learning method [17] to initialize the learningparameters. More specifically, the Q-values are initialized withthe offloading experience in similar environments, i. e., anumber of indoor or outdoor networks with typical MECdeployments given the same energy harvesting source. TheRLO scheme reduces random explorations at the initial stageof the dynamic computation offloading process, and thusaccelerates the learning speed.

In each time slot, according to the EH model such as (6),the IoT device can estimate the amount of harvested energyρ̂(k), observe its current battery level b(k) and the radio linkdata rates to the M edge devices in the last time slot (i.e.,B

(k−1)1 , ..., B

(k−1)M ), which are used to formulate the state

denoted by s(k), with s(k) =[B

(k−1)1 , ..., B

(k−1)M , ρ̂(k), b(k)

].

The IoT device selects edge device i and the rate ofcomputation tasks to offload, i.e., a(k) = [i(k), x(k)], based onstate s(k), and applies the ε-greedy policy with 0 < ε ≤ 1to avoid staying in the local maximum. More specifically,

5

Algorithm 1 RL-based computation offloading scheme

1: Initialize C, f , N , B(0)1 , ..., B

(0)M , ρ̂(1), b(1), α, γ and ε

2: Set Q← Q according to the transfer learning technique3: for k = 1, 2, 3, ... do4: Estimate the amount of harvested energy ρ̂(k)

5: Observe the radio transmission rate, B(k−1)1 , ..., B

(k−1)M ,

and the current battery level b(k)

6: s(k) =[B

(k−1)1 , ..., B

(k−1)M , ρ̂(k), b(k)

]7: Select a(k) = [i(k), x(k)] ∈ A via ε-greedy policy8: Offload x(k)C data to the selected i-th edge device9: Observe the the battery level

10: Evaluate the energy consumption, computation delayand task drop loss

11: Evaluate U (k) via (8)12: Update Q(s(k), a(k)) via (9)13: end for

the offloading policy that achieves the highest expected long-term utility or Q-function denoted by Q(s(k), a) for currentstate is selected with a high probability 1 − ε, while each ofother offloading strategies is selected with a low probabilityε/((Nx + 1)M) to encourage explorations.

The IoT device offloads computation tasks associate withthe x(k)C(k) data to edge device i at time slot k, andlocally processes the remaining tasks. When the edge devicecompletes processing the tasks on behalf of the IoT device,it sends the computation results back to the IoT device. It isassumed that the output of the computation results is of smallsize, thus the transmission delay between the edge device andthe IoT device is negligible. The IoT device then evaluatesthe data sharing gain, measures the total energy consumptionand the results of the processed tasks, and measures thecomputation delay represented by T (k) which depends on thelocal execution delay T

(k)0 and the transmission delay to the

selected edge device T (k)i , i.e., T (k) = max{T (k)

0 , T(k)i }.

The IoT device fails to perform the computation task incase of insufficient energy, i.e., b(k+1) = 0. The task drop lossdenoted by ψ is defined as the cost if a computation task fails.The indicator function denoted by I (ϖ) equals 1 if ϖ is trueand 0 otherwise. Let β and µ be the weighting parameters ofenergy consumption and computation delay of the IoT device,respectively. The utility of the IoT device to select edge devicei at time slot k is denoted by U (k)

i (x), which depends on thedata sharing gains, the task drop loss, the energy consumption,and the overall computation delay, and is given by:

U(k)i (x) = xC(k) − ψI

(b(k+1) = 0

)− βE(k) − µT (k). (8)

The system state transits to the next state s(k+1) if the IoTdevice offloads x(k)C(k) data to the selected i-th edge deviceat state s(k). According to this offloading experience, denotedby

(s(k), a(k), U (k), s(k+1)

), the IoT device updates the Q-

function of the state-action pair (s(k), a(k)) in the following

manner:

Q(

s(k), a(k))← (1− α)Q

(s(k), a(k)

)+ α

(U (k) + γmax

a′∈AQ(

s(k+1), a′))

, (9)

where the learning rate α ∈ (0, 1] is the weight of the currentoffloading experience. The discount factor γ ∈ [0, 1] denotesthe myopic view of the IoT device regarding the future reward.The computation offloading scheme is summarized in Algo-rithm 1. This scheme can achieve the optimal computationoffloading policy via trial-and-error over sufficient time slots.

The learning speed of the RLO scheme increases with thedimension of the action-state space, which increases with thenumber of feasible IoT battery levels, the amount of renewableenergy generated in a time slot, and the number of potentialradio transmission rates corresponding to each edge device,yielding serious performance degradation.

V. DEEP RL-BASED COMPUTATION OFFLOADING

Herein, we propose a deep RL-based offloading schemeDRLO to improve the computation performance. This schemeutilizes a deep CNN to compress the state space and acceleratethe convergence speed of offloading process in Algorithm 1.Fig. 2 illustrates that the deep CNN is a nonlinear approxima-tor of the Q-value for each offloading policy.

Algorithm 2 summarizes the DRLO scheme, which extendsthe system state s(k) as in Algorithm 1 to the offloading expe-rience sequence at time slot k denoted by φ(k), to acceleratethe learning speed and improve the computation performanceof the IoT device. More specifically, the experience sequenceconsists of the current system state s(k) and the previous Wsystem state-action pairs, i.e.,

φ(k) =(

s(k−W ), a(k−W ), ..., s(k−1), a(k−1), s(k)). (10)

If the number of offloading experiences is less than W , i.e.,k ≤W , the IoT device randomly selects an offloading policyfrom action set A, as shown in Algorithm 2.

Fig. 2 shows that the experience sequence, φ(k), is reshapedinto an n0 × n0 square matrix and then input into the CNNwith weights denoted by θ(k). The CNN consists of twoconvolutional (Conv) layers and two fully connected (FC)layers. The first Conv layer consists of m1 different filters.Each filter has size n1×n1 and uses stride s1. The output of thefirst Conv is then passed through a rectified linear unit (ReLU)given in [18] as the activation function. The second Conv layerincludes m2 different filters. Each filter has size n2 × n2 andstride s2, and uses the same linear rectifier. The outputs ofthe 2nd Conv layer are flattened to a vector and then sentto the two FC layers. The first FC layer involves m3 rectifiedlinear units, and the second FC layer provides Q

(φ(k), a; θ(k)

)to estimate the Q-values for all |A| computation offloadingstrategies at the current system sequence φ(k).

To make a tradeoff between exploitation and exploration,the IoT device selects the offloading policy, a(k) ∈ A, basedon the output of the CNN according to the ε-greedy policy[35], and then sends x(k)C(k) data to edge device i.

6

k

ki x

k k-w k-w k- k

d

H

k

k= B

k-BM

k-b

k

xk

Ck

i

dU

d d+

M

B k BMk

k

bk

k

xk

Ck

Uk

n n n n m n2 n2 m

m| |

k

ek k k

Uk k

e U

Ek

Tk

k

Fig. 2. Illustration of the deep RL-based computation offloading scheme for IoT devices.

Algorithm 2 Deep RL-based computation offloading scheme

1: Initialize C, f , N , B(0)1 , ..., B

(0)M , ρ̂(1), b(1), W , H , and

D = ∅2: θ(0) ← θ∗

3: for k = 1, 2, 3, ... do4: Estimate the amount of harvested energy ρ̂(k)

5: Observe the radio transmission rate, B(k−1)1 , ..., B

(k−1)M ,

and the current battery level b(k)

6: s(k) =[B

(k−1)1 , ..., B

(k−1)M , ρ̂(k), b(k)

]7: if k ≤W then8: Select a(k) = [i(k), x(k)] ∈ A at random9: else

10: Form the experience sequence11: φ(k) =

(s(k−W ), a(k−W ), s(k−W+1), ..., s(k)

)12: Set φ(k) as the input of the CNN13: Observe the output of the CNN to obatain

Q(φ(k), a; θ(k)

)14: Select a(k) ∈ A via ε-greedy policy15: end if16: Offload x(k)C data to the selected i-th edge device17: Observe the battery level18: Evaluate the energy consumption, computation delay

and task drop loss19: Evaluate U (k) via (8)20: D ← D

∪(φ(k), a(k), U (k),φ(k+1)

)21: for d = 1, 2, 3, ..., H do22: Select

(φ(d), a(d), U (d),φ(d+1)

)∈ D at random

23: Calculate R via (13)24: end for25: Update the CNN weights with θ(k) by minibatch gra-

dient descent26: end for

After receiving the processed results from the edge device,the IoT device evaluates the data sharing gains, the currentbattery level, the energy consumption and the overall delay toobtain its reward or utility U (k) via (8). The DRLO schemealso updates a quality function Q for each action-state pair inthe dynamic computation offloading process, given by

Q(φ(k), a

)= Eφ′

[U (k) + γmax

a′∈AQ (φ′, a′) |φ(k), a

](11)

where φ′ is the next state sequence based on the selection ofoffloading policy a(k) at state φ(k).

The experience replay utilizes the memory pool as shown inFig. 2 to store the offloading experiences. The offloading expe-rience that the IoT device obtained at time slot k is denoted bye(k) =

(φ(k), a(k), U (k),φ(k+1)

), and the memory pool is D,

with D ={

e(1), ..., e(k)}

. According to the experience replaytechnique, the IoT device randomly selects an experience fromD to update the CNN weight θ(k). The stochastic gradientdescent algorithm is applied, in which the mean-squared errorbetween the network’s output and the target optimal Q-valueis minimized with the minibatch updates. The loss function isgiven by [18]

L(θ(k)) = Eφ,a,U,φ′

[(R−Q

(φ, a; θ(k)

))2]

(12)

where the target optimal Q-function R is given by

R = U + γmaxa′∈A

Q(φ′, a′; θ(k−1)

). (13)

This process repeats H times, and θ(k) is then updatedaccording to these randomly selected experiences.

Similar to Algorithm 1, the transfer learning method isapplied to initialize the CNN parameters in the DRLO schemerather than initializing them randomly to accelerate the learn-

7

ing speed. The IoT device stores emulational experience{φ(k), a(k), U (k),φ(k+1)} in the database E, and the resultingCNN weights θ∗ based on ξ experiences are used to initializeθ(0) as shown in Algorithm 2.

VI. PERFORMANCE EVALUATIONS

The optimal offloading policy is analyzed for a staticscenario, which can be used as the performance upper boundto evaluate the RL-based offloading scheme in the dynamicprocess over sufficient time slots. The time index k in thesuperscript is omitted in the following for clarity. The IoTdevice selects the optimal offloading policy a∗ = [i∗, x∗],with x∗ of computation tasks offloaded to edge device i∗ tomaximize the utility of the IoT device. According to [9] and[14], the IoT device has the same local CPU-cycle frequencyf during the learning process and takes N CPU cycles toperform a single computation task.

Proposition 1. The optimal offloading policy of the IoT deviceis a∗ = [i∗, 1], if

Bi∗ ≥ max

{µ+ βP

βςNf2 + 1,

P

ςNf2

}(14)

where

i∗ = arg max1≤i≤M

Bi. (15)

Proof. By (8), ∀Bi ≤ xf/ ((1− x)N), we have

∂Ui

∂Bi=βP + µ

Bi≥ 0, (16)

and ∀Bi ≥ xf/ ((1− x)N), we have

∂Ui

∂Bi=βP

Bi≥ 0 (17)

indicating that the utility of IoT device, Ui, is maximized ati∗ = argmax1≤i≤M Bi.

By (8) and (14), if NBi∗/ (NBi∗ + f) < x ≤ 1, we have

Ui∗(1) = C

(1− βP + µ

Bi∗

)− ψI

(b− PC

Bi∗+ ρ = 0

)≥

(βςNf2 − βP + µ

Bi∗+ 1

)xC − βςCNf2

− ψI(b+

(ςNf2 − P

Bi∗

)xC − ςCNf2 + ρ = 0

)= Ui∗(x). (18)

Similarly, if 0 ≤ x ≤ NBi∗/ (NBi∗ + f), we have

Ui∗(1) ≥(1 + βςNf2 − βP

Bi∗+µN

f

)xC

− ψI(b+

(ςNf2 − P

Bi∗

)xC − ςCNf2 + ρ = 0

)− βςCNf2 − µCN

f= Ui∗(x). (19)

Thus, we prove a∗ = [i∗, 1] is the optimal offloading policyof the IoT device.

Remark 1: If the IoT device has a good radio channel toat least one edge device as indicated in (14), the IoT device

offloads all the computation tasks to the edge device withthe largest radio link rate as indicated in (15) to maximizeits utility due to the low computation delay and energyconsumption.

In the dynamic computation offloading process, the IoTdevice selects its proportion of data to offload to an edgedevice based on the state of the current system. The nextsystem state observed by the IoT device is independent ofthe previous states and actions. Therefore, the computationoffloading process can be viewed as an MDP.

Theorem 1. If (14) and (15) hold for a sufficient long timeto offload, the RL-based offloading schemes as presented inAlgorithms 1 and 2 can achieve the utility of the IoT devicegiven by

U = C

(1− βP + µ

Bi∗

)− ψI

(b− PC

Bi∗+ ρ = 0

). (20)

Proof. If (14) and (15) hold, we have the optimal offloadingpolicy a∗ = [i∗, 1]. By (1)-(5) and (8), we have (20). Accord-ing to [16], the proposed schemes can eventually derive theoptimal offloading policy of the IoT device a∗ = [i∗, 1] in theMDP via trial-and-error. Therefore, if (14) and (15) hold, theproposed schemes can achieve the utility (20) after a sufficientlong iterative time.

Corollary 1. If (14) and (15) hold for a sufficient long time tooffload, the IoT device with the RL-based offloading schemesas presented in Algorithms 1 and 2 can achieve the computa-tion delay and energy consumption given by T = C/Bi∗ andE = PC/Bi∗ .

Remark 2: The IoT device applies the RL-based offloadingscheme without knowledge of the MEC model and the exactenergy consumption and computation delay model in thedynamic offloading process, which can achieve the optimalcomputation performance. In this scenario, the radio channelbetween the IoT device and edge devices are in a goodcondition. By trail-and-error, the IoT device offloads all thecomputation tasks to the edge device to optimize its per-formance. In this case, the computation delay and energyconsumption are positively linearly related to the generatedtask size C.

Proposition 2. The optimal offloading policy of the IoT deviceis a∗ = [i∗, 0], if

Bi∗ ≤βPf

f + βςNf3 + µN(21)

where

i∗ = arg max1≤i≤M

Bi. (22)

Proof. The proof is similar to Proposition 1.

Remark 3: If all the edge devices experiences severe chan-nel fading as indicated in (21), the IoT device processes allthe computation tasks locally to avoid wasting the transmissionpower.

8

Theorem 2. If (21) and (22) hold for a sufficient long timeto offload, the RL-based offloading schemes given in 1 and 2can achieve the utility of the IoT device given by

U = −ψI(b− ςCNf2 + ρ = 0

)− CN

(βςf2 +

µ

f

).

(23)

Proof. The proof is similar to Theorem 1.

Corollary 2. If (21) and (22) hold for a sufficient long time tooffload, the IoT device with the RL-based offloading schemesas presented in Algorithms 1 and 2 can achieve the computa-tion delay and energy consumption given by T = CN/f andE = ςCNf2.

Remark 4: In this scenario, all the edge devices experiencessevere channel fading. By trial-and-error, the IoT device pro-cesses all the computation tasks locally to maximize its utilitydue to the high transmission delay and energy consumption.

Proposition 3. The optimal offloading policy of the IoT deviceis

a∗ = [i∗,NBi∗

NBi∗ + f] (24)

if

min

{βP + µ

βςNf2 + 1,

P

ςNf2

}≥ Bi∗ ≥

P

ςNf2(25)

where

i∗ = arg max1≤i≤M

Bi. (26)

Proof. The proof is similar to Proposition 1.

Remark 5: If the best edge device experiences modestfading as indicated in (25), the IoT device processes partialcomputation tasks locally and chooses the edge device withthe best link state as indicated in (26).

Theorem 3. If (25) and (26) hold for a sufficient long timeto offload, the RL-based offloading schemes as presented inAlgorithms 1 and 2 can achieve utility of the IoT device givenby

U =− ψI(b+

CN2ςf2Bi∗ − CNPNBi∗ + f

− CNςf2 + ρ = 0

)− CN

(βςf2 +

βςNf2Bi∗ +Bi∗ + µ− βPNBi∗ + f

). (27)

Proof. The proof is similar to Theorem 1.

Corollary 3. If (25) and (26) hold, the IoT device with theRL-based offloading schemes as presented in Algorithms 1 and2 can achieve the computation delay and energy consumptiongiven by

T = max

{CN

f

(1− NBi∗

NBi∗ + f

),

CN

NBi∗ + f

}(28)

E = ςCNf2(1− NBi∗

NBi∗ + f

)+

CNP

NBi∗ + f. (29)

Remark 6: In this case, by trial-and-error, the IoT deviceprocesses partial computation tasks locally and offloads the

remaining computation tasks to the edge device to tradeoffthe energy consumption and computation delay, in order tomaximize its utility in (27). The computation delay and energyconsumption as indicated in (28) and (29) is positively relatedto the generated task size C, and the utility is negatively relatedto C.

The computation complexity of DRLO denoted by Γ mostlyrelies on the CNN structure. Let m0 denote the number ofthe input channels of the CNN, and Ml denote the size ofthe output features of Conv l. According to [36], the size ofthe output features of Conv 1 and that of the Conv 2 are(n0 − n1) /s1 +1 and (n0 − n1) / (s1s2)− (n2 − 1) /s2 +1,respectively. Thus we have the following results:

Theorem 4. The computational complexity of the DRLOscheme is given by

Γ =O(m1n

22m2

(n0 − n1s1s2

− n2 − 1

s2+ 1

)2 ). (30)

Proof. The computational complexity of the FC layers is notinvolved because it is sufficiently small compared with theConv layers. According to the CNN architecture in [18], wehave

m1n22m2

(n0 − n1s1s2

− n2 − 1

s2+ 1

)2

≥ n21m1

(n0 − n1s1

+ 1

)2

.

Thus, according to [37], we have

Γ =O(Σ2

l=1ml−1n2lmlM

2l

)=O

(n21m1

(n0 − n1s1

+ 1

)2

+m1n22m2

(n0 − n1s1s2

− n2 − 1

s2+ 1

)2 )=O

(m1n

22m2

(n0 − n1s1s2

− n2 − 1

s2+ 1

)2 ). (31)

This completes the proof.

Remark 7: The computation complexity of the DRLOincreases with the number of the CNN filters and the filtersize of the Conv layers and decreases with the filter strides.On the other hand, the computation performance of the DRLOimproves with the number of CNN filters that representsthe features of the MEC system and a shorter stride thatcaptures more computation offloading details. The parametersin Algorithm 2 such as the number of filters and filter strideshave to be chosen as a trade-off between the computationalcomplexity and the computation performance.

VII. SIMULATION RESULTS

In this section, the performance of the proposed RL-basedcomputation offloading schemes for the IoT device is evaluatedin the dynamic network with three edge devices. Simulationswere performed for an IoT device powered with RF energyharvester that aims to provide real time emergency care andhealthcare advice. The computation tasks are generated at aspeed of 120 kb/s, with each bit takes 1000 CPU cycles toprocess according to [24]. The IoT device uses the energy

9

TABLE IICNN PARAMETERS

Layer Conv 1 Conv 2 FC 1 FC 2Input 6× 6 4× 4× 20 360 180

Filter size 3× 3 2× 2 / /Stride 1 1 / /

# filters 20 40 180 |A|Activation ReLU ReLU ReLU ReLU

Output 4× 4× 20 3× 3× 40 180 |A|

harvesting model given by (6) to estimate the amount ofthe renewable energy generated by the EH module in thistime slot. The energy harvesting efficiency υ is 0.51, andthe RF transmitter generates 3 W equivalent isotropicallyradiated energy power. The combined gain of the RF energytransmitter antenna and IoT antenna is 6 dB and the path lossis 2. The distance d(k) is modeled as a Markov chain withPr

(d(k+1) = m|d(k) = n

)= dmn, ∀ m,n ∈ {1, 3, 5, 7, 9} m,

according to [21].The estimation error of the IoT device regarding the amount

of harvested energy is assumed to follow a uniform distributionwith △(k) ∼ 0.1U(−1, 1). The IoT device uses the transmitpower 0.5 W, and has the effective capacitance coefficient10−11 according to [23]. The IoT device uniformly quantizesthe battery level 0 ≤ b(k) ≤ 7.4 Wh into 21 levels andthe harvested energy 0 ≤ ρ(k) ≤ 1.7 × 10−3 Wh into 6levels as a tradeoff between the exploration cost and offloadingperformance.

In the simulations, the radio transmission rate of the IoT-edge 1 link is modelled as a Markov chain with Pr(B

(k+1)1 =

m|B(k)1 = n) = Bmn

1 , ∀ m,n ∈ {3, 4, 5, 6, 7} Mbps.Similarly, the radio transmission rates for the 2nd (or 3rd)edge device is modelled as an independent Markov chainwith Pr(B

(k+1)2 = m|B(k)

2 = n) = Bmn2 (or Pr(B

(k+1)3 =

m|B(k)3 = n) = Bmn

3 ), ∀ m,n ∈ {8, 9, 10, 11, 12} Mbps (or{5, 6, 7, 9, 10} Mbps). Unless otherwise specified, a typicaltime slot lasts 1 s, ψ = 10, β = 0.7 and µ = 1.

In the learning algorithm, we set α = 0.9, γ = 0.5,ε = 0.1, W = 10, and H = 16, and the CNN parametersof the DRLO scheme are selected and listed in Table II,to achieve good offloading performance according to theexperiments not presented here. We define the task drop rateas the fraction of failed computation tasks. The simulationsevaluate three benchmark scheme, including DRL in [29],the Q-learning based offloading scheme in [28], and the non-offloading scheme.

Fig. 3 shows that the DRLO scheme achieves the optimalcomputation offloading performance after convergence, whichis in close agreement with the analysis results in Theorem 1.For example, the utility of the IoT device almost convergesto the performance bound, given by (20), if the simulationsettings satisfy (14) and (15). In this case, the IoT devicetends to offload all the computation tasks to the edge deviceto maximize the system performance.

Moreover, the RLO scheme outperforms the Q-learningoffloading scheme, by having a lower energy consumption,

0 500 1000 1500 2000 2500 3000

Time slot

0

0.005

0.01

0.015

0.02

0.025

Ene

rgy

cons

umpt

ion

(J)

Q [28]RLODRLODRL [29]

(a) Energy consumption

0 500 1000 1500 2000 2500 3000

Time slot

15

20

25

30

35

40

45

Com

puta

tion

dela

y (m

s)

Q [28]RLODRLODRL [29]

(b) Computation delay

0 500 1000 1500 2000 2500 3000

Time slot

0

0.1

0.2

0.3

0.4

0.5

0.6

Tas

k dr

op r

ate

Q [28]RLODRLODRL [29]

(c) Task drop rate

0 500 1000 1500 2000 2500 3000

Time slot

-20

0

20

40

60

80

100

120

Util

ity o

f th

e Io

T d

evic

e

Q [28]RLODRLODRL [29]Theoritical value

(d) Utility of the IoT device

Fig. 3. Performance of the IoT computation offloading over time.

10

60 80 100 120 140

Size of the computation task (kb), C

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

Ene

rgy

cons

umpt

ion

(J)

Q [28]RLODRLODRL [29]non-offloading

(a) Energy consumption

60 80 100 120 140

Size of the computation task (kb), C

0

10

20

30

40

50

60

Com

puta

tion

dela

y (m

s)

Q [28]RLODRLODRL [29]non-offloading

(b) Computation delay

60 80 100 120 140

Size of the computation task (kb), C

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Tas

k dr

op r

ate

Q [28]RLODRLODRL [29]non-offloading

(c) Task drop rate

Fig. 4. Average performance of the IoT computation offloading for the givensize of computation task.

shorter computation delay, and lower task drop rate. Forinstance, the RLO scheme reduces the energy consumptionby 28.6%, the computation delay by 16.7%, and the task droprate of the IoT device by 25.0%, compared to the Q-learningscheme at time slot 1000. The computation offloading withDRLO further improves the performance, e.g., it reduces theenergy consumption by 75.0%, computation delay by 32.6%,and the task drop rate of the IoT device by 85.6%, at timeslot 1000, compared to the RLO scheme. That is attributedto the fact that the DRLO algorithm, an extension of Q

1 1.5 2 2.5 3

RF transmission power (W),

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Tas

k dr

op r

ate

Q [28]RLODRLODRL [29]non-offloading

Fig. 5. Task drop rate of the IoT computation offloading for given RFtransmission power, η.

learning, compresses the learning state space by using CNN toaccelerate the learning process and enhance the computationoffloading performance. If the interaction time is sufficientlylong enough, the RLO scheme can also converge to the optimaltheoretical results. We can also see that, the proposed DRLOscheme outperforms the DRL scheme in [29]. For instance, itreduces the energy consumption by 58.3%, the computationdelay by 26.7%, the task drop rate of the IoT device by55.5%, and thus increases the utility by 14.6%, at time slot1000. That is because the DRLO scheme incorporates the EHtechnique and more system state to improve the computationperformance.

The offloading performance averaged over first 1500 timeslot in the dynamic offloading is studied. Fig. 4 shows therelationship between the system performance and the size ofthe computation task C. Clearly, the energy consumption,computation delay, and task drop rate of the IoT deviceincrease with the amount of computation task in MEC. Forinstance, if the amount of the computation task changes from60 kb to 140 kb, the energy consumption, computation delayand task drop rate of the IoT device with DRLO schemeincrease by 33.3%, 58.5% and 81.8%, respectively. In thedynamic game with C = 100 kb computation tasks, the DRLOoutperforms the RLO with 68.1% lower energy consumption,33.3% shorter computation delay, and 75.7% lower task droprate, and outperforms the non-offloading scheme with 89.1%lower energy consumption, 62.5% shorter computation delay,and 88.1% lower task drop rate.

Fig. 5 exhibits the impact of the RF transmission power η ofthe energy transmitter in the wireless powered MEC network,with η increases from 1 W to 3 W. The task drop rate decreaseswith the increase in the RF transmission power of the energytransmitter. For instance, if the RF transmission power changesfrom 1 W to 3 W, the task drop rate of the DRLO schemedecreases by 90.8%. In the dynamic mobile offloading processwith the RF transmission power η = 2 W, the task drop rateof the DRLO is 65.7% less than that of RLO, which is 58.6%and 90.1% lower than that of the DRL and non-offloadingscheme, respectively.

As shown in Table III, the DRLO requires 345 MB memory

11

TABLE IIICOMPUTATION OVERHEAD OF THE RL-BASED OFFLOADING SCHEMES

SchemeMemory

size(MB)

Policyselectiontime (ms)

Convergencytime

(time slot)DRLO 345 8.3 400RLO 168 0.4 > 3000

most of the time, which can be supported by IoT platforms,such as Qualcomm Snapdragon 800 and Nvidia Tegra K1according to [19]. On the other hand, the IoT devices withconstrained computational resources that cannot support deeplearning can use RLO to select the offloading policy withsignificantly reduced computation complexity. For example,the RLO scheme saves 95.2% of the time to select theoffloading policies in a time slot compared with the DRLO.

VIII. CONCLUSION

Energy harvesting powered mobile edge computing is apromising approach to improve the computation capabilitiesfor self-sustainable IoT devices. In this paper, we have pre-sented the RL-based computation offloading framework forIoT devices with EH to achieve the optimal offloading policywithout awareness of the MEC model, the computation delayand energy consumption model. An RLO scheme has beenproposed for IoT devices with low complexity, which utilizesthe transfer learning technique to accelerate the learning speed.We have also presented a DRLO scheme that uses CNN tocompress the state space in the learning process. Furthermore,the conditions for both “fully offloading” and “locally process-ing” are provided, and the performance bounds of the proposedRL-based offloading after convergence under three typicalscenarios are studied. Simulations have been performed forthe IoT device with RF signal-based wireless power transferto verify the theoretical results, indicating that the DRLOscheme generates the best policy, and outperforms the DRLscheme with 58.3% lower energy consumption, 26.7% lesscomputation delay, and 55.5% lower task drop rate. We willimplement the RL-based offloading schemes and evaluate theirperformance via experiments in our future work.

REFERENCES

[1] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A surveyon mobile edge computing: The communication perspective,” IEEECommun. Surveys and Tutorials, vol. 19, no. 4, pp. 2322 – 2358, Aug.2017.

[2] X. Hou, Y. Li, M. Chen, D. Wu, D. Jin, and S. Chen, “Vehicular fogcomputing: A viewpoint of vehicles as the infrastructures,” IEEE Trans.Vehicular Technology, vol. 65, no. 6, pp. 3860–3873, Jun. 2016.

[3] J. Feng, Z. Liu, C. Wu, and Y. Ji, “Ave: Autonomous vehicularedge computing framework with aco-based scheduling,” IEEE Trans.Vehicular Technology, vol. 66, no. 12, pp. 10660–10675, Dec. 2017.

[4] D. Huang, P. Wang, and D. Niyato, “A dynamic offloading algorithmfor mobile computing,” IEEE Trans. Wireless Commun., vol. 11, no. 6,pp. 1991–1995, Jun. 2012.

[5] M. Chiang and T. Zhang, “Fog and IoT: An overview of researchopportunities,” IEEE Internet of Things J., vol. 3, no. 6, pp. 854–864,Jun. 2016.

[6] S. Sudevalayam and P. Kulkarni, “Energy harvesting sensor nodes:Survey and implications,” IEEE Commun. Surveys and Tutorials, vol. 13,pp. 443–461, Jul. 2011.

[7] D. Mishra, S. De, S. Jana, et al., “Smart RF energy harvesting commu-nications: Challenges and opportunities,” IEEE Commun. Mag., vol. 53,no. 4, pp. 70–78, Apr. 2015.

[8] D. Mishra, S. De, and D. Krishnaswamy, “Dilemma at RF energy har-vesting relay: Downlink energy relaying or uplink information transfer?,”IEEE Trans. Wireless Commun., vol. 16, no. 8, pp. 4939–4955, Aug.2017.

[9] F. Wang, J. Xu, X. Wang, and S. Cui, “Joint offloading and computingoptimization in wireless powered mobile-edge computing systems,”IEEE Trans. Wireless Commun., vol. 17, no. 3, pp. 1784 – 1797, Dec2017.

[10] T. Q. Dinh, J. Tang, Q. D. La, and T. Q. Quek, “Offloading in mobileedge computing: Task allocation and computational frequency scaling,”IEEE Trans. Commun., vol. 65, no. 8, pp. 3571–3584, Apr. 2017.

[11] C. You, K. Huang, and H. Chae, “Energy efficient mobile cloudcomputing powered by wireless energy transfer,” IEEE J. Sel. AreasCommun., vol. 34, no. 5, pp. 1757–1771, May 2016.

[12] S. Bi, Y. Jun, et al., “Computation rate maximization for wirelesspowered mobile-edge computing with binary computation offloading,”CoRR, vol. abs/1708.08810, Aug. 2017. [Online]. Available: http://arxiv.org/abs/1708.08810.

[13] F. Lin, C. Chen, N. Zhang, X. Guan, and X. Shen, “Autonomous channelswitching: Towards efficient spectrum sharing for industrial wirelesssensor networks,” IEEE Internet of Things J., vol. 3, no. 2, pp. 231–243, Apr. 2016.

[14] Y. Mao, J. Zhang, and K. B. Letaief, “Dynamic computation offloadingfor mobile-edge computing with energy harvesting devices,” IEEE J.Sel. Areas Commun., vol. 34, no. 12, pp. 3590–3605, Sept. 2016.

[15] J. Xu, L. Chen, and S. Ren, “Online learning for offloading andautoscaling in energy harvesting mobile edge computing,” IEEE Trans.Cognitive Commun. and Networking, vol. 3, no. 3, pp. 361 – 373, Jul.2017.

[16] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction.MIT press, Cambridge, MA, 1998.

[17] H. Zuo, G. Zhang, W. Pedrycz, V. Behbood, and J. Lu, “Fuzzy regressiontransfer learning in takagi–sugeno fuzzy models,” IEEE Trans. FuzzySyst., vol. 25, no. 6, pp. 1795–1807, Dec. 2017.

[18] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, et al.,“Human-level control through deep reinforcement learning,” Nature,vol. 518, no. 7540, pp. 529–533, Jan. 2015.

[19] N. D. Lane, S. Bhattacharya, P. Georgiev, et al., “An early resource char-acterization of deep learning on wearables, smartphones and internet-of-things devices,” in Proc. Int. workshop on internet of things towardsapplications, pp. 7–12, New York, NY, Nov. 2015.

[20] N. D. Lane, P. Georgiev, and L. Qendro, “Deepear: robust smartphoneaudio sensing in unconstrained acoustic environments using deep learn-ing,” in Proc. 2015 ACM Int. Joint Conf. Pervasive and UbiquitousComputing, pp. 283–294, Osaka, Japan, Sept. 2015.

[21] S. Bi, C. K. Ho, and R. Zhang, “Wireless powered communication:Opportunities and challenges,” IEEE Commun. Mag., vol. 53, no. 4,pp. 117–125, Apr. 2015.

[22] X. Lu, P. Wang, D. Niyato, D. I. Kim, and Z. Han, “Wireless networkswith RF energy harvesting: A contemporary survey,” IEEE Commun.Surveys and Tutorials, vol. 17, no. 2, pp. 757–789, Nov. 2015.

[23] W. Zhang, Y. Wen, K. Guan, D. Kilper, H. Luo, and D. O. Wu, “Energy-optimal mobile cloud computing under stochastic wireless channel,”IEEE Trans. Wireless Commun., vol. 12, no. 9, pp. 4569–4581, Aug.2013.

[24] C. You, K. Huang, H. Chae, and B. H. Kim, “Energy-efficient resourceallocation for mobile-edge computation offloading,” IEEE Trans. Wire-less Commun., vol. 16, no. 3, pp. 1397–1411, Mar. 2017.

[25] Y. Wang, M. Sheng, X. Wang, et al., “Mobile-edge computing: Partialcomputation offloading using dynamic voltage scaling,” IEEE Trans.Commun., vol. 64, no. 10, pp. 4268–4282, Aug. 2016.

[26] L. Xiao, C. Xie, T. Chen, H. Dai, and H. V. Poor, “A mobile offloadinggame against smart attacks,” IEEE Access, vol. 4, pp. 2281–2291, May2016.

[27] W. Zhang, Y. Hu, Y. Zhang, and D. Raychaudhuri, “Segue: Quality ofservice aware edge cloud service migration,” in Proc. IEEE Int. Conf.Cloud Computing Technol. and Sci., pp. 344–351, Luxembourg City,Luxembourg, Dec. 2016.

[28] L. Xiao, Y. Li, X. Huang, and X. Du, “Cloud-based malware detectiongame for mobile devices with offloading,” IEEE Trans. Mobile Comput-ing, vol. 16, no. 10, pp. 2742–2750, Oct. 2017.

[29] J. Li, H. Gao, T. Lv, et al., “Deep reinforcement learning basedcomputation offloading and resource allocation for MEC,” in Proc. IEEE

12

Wireless Commun. and Networking Conf. (WCNC), pp. 1–6, Barcelona,Spain, Jun. 2018.

[30] N. Lu, N. Cheng, N. Zhang, X. Shen, and J. W. Mark, “Connectedvehicles: Solutions and challenges,” IEEE Internet of Things J., vol. 1,no. 4, pp. 289–299, May 2014.

[31] O. Munoz, A. Pascual-Iserte, and J. Vidal, “Optimization of radio andcomputational resources for energy efficiency in latency-constrainedapplication offloading,” IEEE Trans. Veh. Technol., vol. 64, no. 10,pp. 4738–4755, Oct. 2015.

[32] W. Liu, J. Cao, L. Yang, L. Xu, X. Qiu, and J. Li, “Appbooster: Boostingthe performance of interactive mobile applications with computationoffloading and parameter tuning,” IEEE Trans. Parallel. Distrib. Syst.,vol. 28, no. 6, pp. 1593–1606, Nov. 2017.

[33] X. Chen, L. Jiao, W. Li, and X. Fu, “Efficient multi-user computationoffloading for mobile-edge cloud computing,” IEEE/ACM Trans. Net-working, vol. 24, no. 5, pp. 2795–2808, Oct. 2016.

[34] H. Wang and J. Huang, “Incentivizing energy trading for interconnectedmicrogrids,” IEEE Trans. Smart Grid, Oct. 2016. [Online]. Available:https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7442079.

[35] E. Rodrigues Gomes and R. Kowalczyk, “Dynamic analysis of multia-gent Q-learning with ε-greedy exploration,” in Proc. ACM Annual Int’l.Conf. Mach. Learn. (ICML), pp. 369–376, Montreal, Canada, Jun. 2009.

[36] C. C. T. Mendes, V. Frémont, and D. F. Wolf, “Exploiting fully convo-lutional neural networks for fast road detection,” in Proc. IEEE Intern.Conf. Robotics and Automation (ICRA), pp. 3174–3179, Stockholm,Sweden, May 2016.

[37] K. He and J. Sun, “Convolutional neural networks at constrained timecost,” in Proc. IEEE conf. computer vision and pattern recognition,pp. 5353–5360, Massachusetts, Boston, Jun. 2015.

Minghui Min received the B.S. degree in automa-tion from QuFu Normal University, Rizhao, China,in 2013, the M.S. degree in control theory and con-trol engineering from Shenyang Ligong University,joint training with, Shenyang Institute of Automa-tion, Chinese Academy of Sciences, Shengyang,China, in 2016. She is currently pursuing the Ph.D.degree with the Department of Communication En-gineering, Xiamen University, Xiamen, China. Herresearch interests include network security and wire-less communications.

Liang Xiao (M’09, SM’13) is currently a Professorin the Department of Communication Engineering,Xiamen University, Fujian, China. She has servedas an associate editor of IEEE Trans. InformationForensics and Security and guest editor of IEEEJournal of Selected Topics in Signal Processing.She is the recipient of the best paper award for2016 INFOCOM Big Security WS and 2017 ICC.She received the B.S. degree in communicationengineering from Nanjing University of Posts andTelecommunications, China, in 2000, the M.S. de-

gree in electrical engineering from Tsinghua University, China, in 2003, andthe Ph.D. degree in electrical engineering from Rutgers University, NJ, in2009. She was a visiting professor with Princeton University, Virginia Tech,and University of Maryland, College Park.

Ye Chen received the B.S. degree in communica-tion engineering from Xiamen University, Xiamen,China, in 2017, where he is currently pursuing theM.S. degree with the Department of CommunicationEngineering, Xiamen University, Xiamen, China.His research interests include network security andwireless communications.

PLACEPHOTOHERE

Peng Cheng (M’10) received the

Di Wu (M’06-SM’17) received the B.S. degree fromthe University of Science and Technology of China,Hefei, China, in 2000, the M.S. degree from the In-stitute of Computing Technology, Chinese Academyof Sciences, Beijing, China, in 2003, and the Ph.D.degree in computer science and engineering fromthe Chinese University of Hong Kong, Hong Kong,in 2007. He was a Postdoctoral Researcher withthe Department of Computer Science and Engineer-ing, Polytechnic Institute of New York University,Brooklyn, NY, USA, from 2007 to 2009, advised by

Prof. K. W. Ross. Dr. Wu is currently a Professor and the Assistant Deanof the School of Data and Computer Science with Sun Yat-sen University,Guangzhou, China. He was the recipient of the IEEE INFOCOM 2009 Bestpaper Award. His research interests include cloud computing, multimediacommunication, Internet measurement, and network security.

Weihua Zhuang (M’93-SM’01-F’08) has been withthe Department of Electrical and Computer Engi-neering, University of Waterloo, Canada, since 1993,where she is a Professor and a Tier I Canada Re-search Chair in Wireless Communication Networks.She is the recipient of 2017 Technical RecognitionAward from IEEE Communications Society Ad Hoc& Sensor Networks Technical Committee, one of2017 ten N2Women (Stars in Computer Network-ing and Communications), and a co-recipient ofseveral best paper awards from IEEE conferences.

Dr. Zhuang was the Editor-in-Chief of IEEE Transactions on VehicularTechnology (2007-2013), Technical Program Chair/Co-Chair of IEEE VTCFall 2017 and Fall 2016, and the Technical Program Symposia Chair of theIEEE Globecom 2011. She is a Fellow of the IEEE, the Royal Society ofCanada, the Canadian Academy of Engineering, and the Engineering Instituteof Canada. Dr. Zhuang is an elected member in the Board of Governors andVP Publications of the IEEE Vehicular Technology Society. She was an IEEECommunications Society Distinguished Lecturer (2008-2011).