Optimization-driven Deep Reinforcement Learningfor Robust Beamforming in IRS-assisted Wireless

CommunicationsJiaye Lin∗, Yuze Zou†, Xiaoru Dong∗, Shimin Gong∗, Dinh Thai Hoang‡ and Dusit Niyato§

∗School of Intelligent Systems Engineering, Sun Yat-sen University, China†School of Electronic Information and Communications, Huazhong University of Science and Technology, China

‡School of Electrical and Data Engineering, University of Technology Sydney, Australia§School of Computer Science and Engineering, Nanyang Technological University, Singapore

Abstract—Intelligent reflecting surface (IRS) is a promisingtechnology to assist downlink information transmissions from amulti-antenna access point (AP) to a receiver. In this paper, weminimize the AP’s transmit power by a joint optimization of theAP’s active beamforming and the IRS’s passive beamforming.Due to uncertain channel conditions, we formulate a robustpower minimization problem subject to the receiver’s signal-to-noise ratio (SNR) requirement and the IRS’s power budgetconstraint. We propose a deep reinforcement learning (DRL)approach that can adapt the beamforming strategies from pastexperiences. To improve the learning performance, we derive aconvex approximation as a lower bound on the robust problem,which is integrated into the DRL framework and thus promotinga novel optimization-driven deep deterministic policy gradient(DDPG) approach. In particular, when the DDPG algorithmgenerates a part of the action (e.g., passive beamforming), we canuse the model-based convex approximation to optimize the otherpart (e.g., active beamforming) of the action more efficiently.Our simulation results demonstrate that the optimization-drivenDDPG algorithm can improve both the learning rate and rewardperformance significantly compared to the conventional model-free DDPG algorithm.

I. INTRODUCTIONRecently, intelligent reflecting surface (IRS) has been in-

troduced as a promising technology to improve energy- andspectrum-efficiency of wireless communications [1]. It iscomposed of a large array of passive scattering elementsinterconnected and individually controlled by an embeddedIRS controller. The joint control of the complex reflectingcoefficients for all scattering elements, i.e., passive beam-forming, can enhance the signal strength to the receiver [2].The IRS’s passive beamforming along with the transceivers’transmission control are envisioned to further improve thenetwork performance. An extensive survey in [3] reveals thatthe IRS has already been applied in diverse network scenarios,serving different roles in wireless communications such as theambient reflector, signal transmitter and even the receiver.

The use of IRS mainly aims at improving the transmissionperformance in terms of signal-to-noise ratio (SNR) or spectralefficiency [4], power consumption or energy efficiency [5], andsecurity enhancement [6]. The IRS can also be used to enhancewireless power transfer [7], mobile edge computing [8], andvehicular communications [9]. The performance maximizationof IRS-assisted wireless systems is typically formulated as a

joint optimization problem of the active and passive beam-forming strategies, e.g., [4]–[8]. However, due to the non-convex problem structure, the solution methods are typicallybased on the alternating optimization (AO) framework withguaranteed convergence to sub-optimal solutions. Within eachiteration of the AO framework, semidefinite relaxation (SDR)or convex approximation are usually required to optimizeeither the active or passive beamforming. As a heuristicapproach, the performance loss of the AO method can notbe known exactly and difficult to characterize precisely [5].Besides, the optimization methods also suffer from a fewpractical difficulties. Firstly, the computational complexity ofthe AO method may increase significantly as the size of IRS’sscattering elements becomes large. This makes it difficultfor practical implementation in a dynamic radio environment.Secondly, a tractable formulation of beamforming optimizationis in fact based on an inexact system modeling, which can bea simplification of the real system, for example, with perfectchannel information, continuous and exact phase control.The problem reformulation or approximation further lead toa deviated solution far from the optimum. In general, wecan expect that the model-based optimization methods onlyprovide a lower performance bound of the original problem.

Different from the optimization methods, machine learningapproaches are more robust against the uncertain systemmodels and have also been applied to IRS-assisted wirelesssystems to realize the IRS’s phase control. The authors in [10]employ a deep neural network (DNN) to map the receiver’slocation to the IRS’s optimal phase configuration in a complexindoor environment. Similarly, the authors in [11]–[13] usewell-trained DNNs to make real-time predictions for the IRS’spassive beamforming, which can achieve close-to-optimal per-formance with reduced time consumption or computationalcomplexity compared to the SDR-based optimization methods.However, the offline training of DNNs relies on either anexhaustive search or the AO methods. The authors in [14]apply the deep deterministic policy gradient (DDPG) algo-rithm to maximize the received SNR of an IRS-assisted systemby continuously interacting with the environment. The DRLapproach is used in [15] to enhance secrecy rate against mul-tiple eavesdroppers. The authors in [16] implement the DRLagent at the IRS, which can observe the channel conditions

arX

iv:2

005.

1188

5v1

[ee

ss.S

P] 2

5 M

ay 2

020

2

and take actions based on the receiver’s feedback. Though theDRL approach can learn the optimal strategy from scratch, itgenerally has a slow learning rate to converge.

In this paper, we propose a novel DRL approach withenhanced learning efficiency to minimize the transmit power ofthe access point (AP) in an IRS-assisted multiple-input single-output (MISO) system with uncertain channel conditions. Wefirstly formulate a robust power minimization problem byjointly optimizing the active and passive beamforming, andthen we construct a Markov decision process (MDP) to solveit by learning from past experiences. To improve learningefficiency, we design the optimization-driven DDPG algorithmthat integrates the model-based optimization into the frame-work of a model-free DDPG algorithm. In particular, whenthe DDPG algorithm generates a part of the action, a model-based optimization module can be used to find the other part ofthe action very efficiently. By solving an approximate convexproblem, the optimization module also provides a achievablelower bound on the original robust problem, which guidesthe DDPG algorithm to search for an optimal action moreefficiently. Our simulation results reveal that the optimization-driven DDPG algorithm not only speeds up the learningrate but also reduces the AP’s transmit power significantlycompared to the conventional DDPG algorithm.

II. SYSTEM MODEL

As shown in Fig. 1, the IRS with N reflecting elementsassists the information transmissions from the M -antenna APto the single-antenna receiver. The AP-receiver, AP-IRS andIRS-receiver complex channels are denoted by g ∈ CM×1,H ∈ CM×N and f ∈ CN×1, respectively. We assume thateach reflecting element can set a phase shift θn ∈ [0, 2π] andits magnitude ρn ∈ [0, 1] to reflect the incident RF signals.

A. SNR and Energy Budget

Let Θ = diag(ρ1ejθ1 , . . . , ρNejθN ) denote the IRS’s pas-sive beamforming, where diag(·) denoting the diagonal matrixgiven the diagonal vector. Hence the IRS-assisted equivalentchannel from the AP to the receiver is given by ĝ = g+HΘf ,where H = [h1, . . . ,hN ] denotes the channel matrix from theAP to the IRS. Let w ∈ CM×1 denote the AP’s beamformingvector and s be the complex symbol with unit transmit power.The received signal at the receiver is thus y = ĝHws + νd,where the superscript denotes the conjugate transpose and νddenotes the Gaussian noise with zero mean and normalizedvariance. Therefore, the received SNR can be characterized as

γ(w,Θ) = ‖(g + HΘf)Hw‖2, (1)

which depends on the AP’s active beamforming w and theIRS’s passive beamforming Θ.

We assume that each tunable chip of the reflecting elementis also equipped with an energy harvester that is able to harvestRF energy from the AP’s beamforming signals. By tuningthe magnitudes of reflecting coefficients ρ , [ρ1, . . . , ρN ]T , apart of the incident signal power is reflected to the receiver,while the other part is fed to the energy harvester. Hence,the parameter ρ is also called as the power-splitting (PS)

Access Point (AP)

Active beamforming

Receiver

𝐠

𝐇

𝐟

Passive beamforming

IRS controller

Energy storageIRS

…𝑑 𝑑

𝑑

Fig. 1: IRS-assisted MISO system.

ratio. To maintain the IRS’s operations, the total harvestedenergy has to meet the IRS’s total power consumption. That is,η∑n(1− ρ2n)‖hHn w‖2 ≥ Nµ, where η represents the power

harvesting efficiency and hn denotes the channel from the APto the n-th reflecting element. The IRS’s power consumptionis given by Nµ, where µ is the power consumption of a singlescattering element [17].

B. Channel Uncertainty Model

We assume that the direct channel g from the AP to thereceiver can be estimated accurately by the active receiver. Inparticular, the HAP can send a known pilot information tothe receiver with fixed transmit power. Meanwhile, the IRSswitches off its reflecting elements. The channel g can berecovered at the receiver based on the received signal samples.

However, by using the passive scattering elements, thechannels H and f have to be estimated at either the HAPor the receiver by overhearing the channel response. Weassume that the channel H is subject to estimation errors,i.e., H = H̄ + ∆h, where H̄ denotes the averaged estimateand ∆h denotes the error estimate of the channel from theHAP to the IRS. The error estimate ∆h has limited powerdensity, and thus we can define the uncertainty set Uh for theAP-IRS channel H as follows:

H ∈ Uh , {H = H̄ + ∆h : Tr(∆Hh ∆h) ≤ δ2h}, (2)

where Tr(·) represents the trace operation and δh denotesthe power limit of error estimate ∆h. The estimation of theIRS-receiver channel f has to be bundled with the AP-IRSchannel H and performed at the receiver by overhearing themixture of signals from the HAP and the IRS’s reflections.By rewriting the cascaded AP-IRS-receiver channel as Hf ,diag(f)H = [f1h1, f2h2, . . . , fNhN ], similar to (2) we candefine the uncertainty for Hf as follows:

Hf ∈ Uf , {Hf = H̄f + ∆f : Tr(∆Hf ∆f ) ≤ δ2f }, (3)

where δf denotes the power limit of the error estimate ∆f forthe reflecting channel Hf . The average channel estimate H̄fand the power limit δf are assumed to be known in advanceby channel measurements.

3

III. ROBUST ACTIVE AND PASSIVE BEAMFORMINGWe aim to minimize the AP’s transmit power, denoted as

||w||2, by jointly optimizing the active and passive beamform-ing strategies, subject to the IRS’s power budget constraintand the receiver’s SNR requirement. Considering a practicalcase that all the reflecting elements have the same PS ratioρ, we can decompose the optimization of ρ and the phasevector θ = [ejθ1 , . . . , ejθN ]T . As such, we can rewrite theIRS-assisted channel by ĝ = g + ρHfθ and then simplifyrobust power minimization problem as follows:

minw,θ,ρ

‖w‖2 (4a)

s.t. |(g + ρHfθ)Hw|2 ≥ γ1, ∀Hf ∈ Uf , (4b)η(1− ρ2)‖HHw‖2 ≥ Nµ, ∀H ∈ Uh, (4c)ρ ∈ (0, 1) and θn ∈ (0, 2π) ∀n ∈ N . (4d)

The first difficulty of the non-convex problem (4) lies in thatthe PS ratio ρ is coupled with the phase vector θ. Anotherdifficulty comes from the semi-infinite constraints in (4b)-(4c), which have to hold for any channel error estimate in theuncertainty set. In the sequel, we consider solving problem (4)by using a learning-based and model-free approach that cantolerate inaccuracies in modeling and online decision-making.

A. Deep Reinforcement Learning Approach

Deep reinforcement learning (DRL) is a combination ofdeep neural networks (DNNs) and reinforcement learning(RL). It aims at solving MDP problems with large action andstate spaces that are difficult to handle by conventional RL ap-proaches [18]. The MDP framework can be defined by a tuple{S,A,P,R}. S represents the system state space denotingthe set of observations of the network environment. A denotesthe set of actions. The state transition probability P denotesthe distribution of the next state st+1 ∈ S given the actionat ∈ A taken in the current state st. It is typically uncertainto the agent and has to be learnt during the decision makingprocess. The immediate reward R : S × A → R provides thequality evaluation rt(st,at) of the state-action pair (st,at). Italso drives the search for the best policy to maximize the long-term reward V (s) = E [

∑∞t=0 γ

trt(st,at)|s], where γ ∈ [0, 1]denotes the discount factor.

1) MDP Reformulation: The most straightforward DRL so-lution to problem (4) is to design a DRL agent at the AP, whichjointly decides the AP’s transmit beamforming and the IRS’spassive beamforming strategies, based on the observed statest ∈ S and the knowledge learnt from past experience Ht ,{. . . , st−1,at−1, rt−1, st}. The system state st = (ct,ot)includes the channel information and the indicator of outageevents in the past signal transmission periods. The channelinformation is denoted by the set ct , {gt,Ht,Hf ,t}t∈T ,where T , {t − T + 1, . . . , t − 1, t} denotes T consecutivepast transmission periods. We can easily extract the averagedchannel estimates H̄ and H̄f from set ct by averaging overT consecutive channel estimations. The power limit δh (orδf ) of the error estimate corresponding to the channel H (orHf ) can be also obtained similarly. In the system state, wealso record the outage events ot , [o1, o2, . . . , oT ]T in the

past T transmission periods. For each t ∈ T , let ot = 1denote an outage event if (4b) or (4c) does not hold, whichimplies the infeasibility of problem (4). Given the currentstate st, the action at , (wt,θt, ρt) includes the AP’sactive beamforming wt and the IRS’s passive beamforming,characterized by (ρt,θt), which are both continuous decisionvariables. The AP’s transmit power will be wasted once outageevents happen. Hence, we can define the immediate reward asthe energy efficiency, i.e., the successfully transmitted dataover the AP’s power consumption:

rt(st,at) = E[(1− ot)|(gt + ρHf ,tθt)Hwt|2||wt||−2]. (5)

The expectation is taken over the past T transmission periods.It is clear that the reward is inversely proportional to the AP’stransmit power when the inequalities hold in (4b)-(4c), andotherwise the reward becomes zero when outage happens.

2) DDPG for Continuous Control: RL provides a solutionto find the optimal policy π∗ : S → A that maps each statest ∈ S to an action at ∈ A such that the value function V (s)is maximized. With small and finite state and action spaces,the optimal policy can be obtained by the Q-learning algo-rithm [18], i.e., the optimal action a∗t = arg maxa∈AQ(st,a)on each state is to maximize the Q-value function, and thenwe update the Q-value by the difference between the currentQ-value and its target value yt:

Qt+1(st,at) = Qt(st,at) + τt

[yt −Qt(st,at)

],

where τt is the step-size and the target yt is evaluated by

yt = rt(st,at) + γmaxat+1

Qt(st+1,at+1). (6)

For a small size of discrete state and action spaces, the Q-valuefor each state and action can be stored in a table and updatedin each decision epoch. However, the Q-learning algorithmbecomes unstable when the state and action spaces are verylarge [19]. Instead, the deep Q Network (DQN) algorithm usesDNN with the weight parameter ωt as the approximator forthe Q-value function. The input to the DNN is the currentstate st and output is the expected action at. The DNNparameter ωt is updated in each decision epoch to output abetter approximation of the Q-value. This can be achieved bytraining the DNN to minimize the loss function:

`(ωt) = E[(yi −Qt(si,ai|ωt))2

]. (7)

One of the main advantages of using DRL lies in that we canlearn from past experiences efficiently by using a group ofhistorical transition samples (si,ai, ri, si+1) ∈Mt, namely, amini-batchMt, to train the DNN at each decision epoch. Theexpectation in (7) is taken over all samples in the mini-batchMt and the target yi is evaluated by (6) for each sample.

The DQN algorithm can be extended to solve optimiza-tion problems in a continuous action space. Besides a DNNapproximator for the Q-value function, the DDPG algorithmalso uses a DNN with the parameter v to approximate thepolicy function [20]. The DNN training aims at updating theparametric policy πv directly in a gradient direction to improvethe estimation of value function, which is given as follows:

J(v) =∑s∈S

d(s)∑a∈A

πv(a|s)Q(s,a|ω), (8)

4

where d(s) denotes the stationary state distribution corre-sponding to the policy πv and Q(s,a|ω) is the Q-valueapproximated by the DNN with parameter ω. By deterministicpolicy gradient theorem [18], the gradient of (8) is given by

∇vJ(v) = Es∼d(s)[∇aQ(s,a|ω)∇vπv(s)|a=πv(s)], (9)

which can be performed efficiently by sampling the historicaltrajectories. The policy gradient in (9) motivates the actor-critic framework, which updates two sets of DNN parameters(v,ω) separately. The actor network updates the policy pa-rameter v in gradient direction as follows:

vt+1 = vt + αv∇aQ(st,at|ωt)∇vπv(s)|at=πv(s).

The critic network updates the Q-network as follows:

ωt+1 = ωt + αωδt∇ωQ(st,at|ωt),

where δt = yt −Q(st,at|ωt) denotes the temporal-difference(TD) error between Q(st,at|ωt) and its target value yt. Twoconstants αv and αω are viewed as step-sizes. The trainingof the critic network is also performed by sampling a mini-batch from the experience replay memory. To ensure betterconvergence, the target yi for critic network is given by

yi = ri(si,ai) + γQ(si+1, π(si+1|v′t)|ω′t). (10)

Here the DNN parameters (v′t,ω′t) of the target networks are

delayed copy of (vt,ωt) from the online networks. Given thecurrent state si, the action is determined by the policy πv′t andthen the Q-value is estimated by the DNN with parameter ω′t.

B. Optimization-driven Learning Strategy

The conventional DDPG approach estimates the target valueyt by the immediate reward rt(st,at) and the target Q-networkwith the parameter ω′t, as shown in (10). To ensure betterconvergence performance, the target Q-network is evolvingfrom the online Q-network by the following rule:

ω′t+1 = τωt + (1− τ)ω′t, (11)

where ωt denotes the DNN parameter of the online Q-networkand τ is a step size. This implies strong coupling between thetwo Q-networks and may lead to slow learning rate. The maindrawbacks can be understood from the following aspects.• Random initialization: Both the online and target Q-

networks can be initialized randomly. In the early stageof learning, the DNN approximations of Q-value can befar from its optimum and thus probably misleading thelearning process. Hence, the DDPG algorithm practicallyrequires a long warm-up period to train both Q-networks.

• Inaccurate reward estimation: The evaluation of the im-mediate reward rt(st,at) is based on the output of thecritic network with non-optimal parameter, especiallyin the early stage of learning. The inaccurate rewardestimation can be also far from its real value.

• Sensitive parameter setting: The choice of parameter τto update ω′t+1 in (11) is problematic. A small value ofτ can stabilize but also slow down the learning, while alarge value of τ implies strong correlation between the

online and target Q-networks, which may result in thefluctuations of learning performance or even divergence.

In this following part, we aim to stabilize and speed up thelearning process by estimating the target value yt in a better-informed and independent way.

1) Merge Model-free and Model-based Estimations: Basedon the DDPG framework, in the t-th decision epoch the actornetwork outputs the action at = (ρt,wt,θt) and the targetQ-network produces an estimation of the target yt, which canbe very far from its optimum in the early stage of learning.This motivates us to use a model-based optimization method toestimate a lower bound on the target value yt based on partialsystem information. Specifically, we can divide the action atinto two parts, i.e., the scalar ρt and two vectors (w,θ).Given the PS ratio ρt, we can solve the robust problem (4) byoptimizing the active and passive beamforming (w,θ). Thissignificantly simplifies the solution to problem (4), and hencewe can easily find a lower bound on the target value yt. Lety′t denote the lower bound on the target value and (w

′,θ′)be the optimized beamforming solution. We envision that themodel-based optimization can provide a better-informed target,i.e., y′t > yt, especially in the early stage of learning.

The flow chart of the proposed optimization-driven DDPGalgorithm is shown in Fig. 2. The actor and critic networksfirstly generate the action and value estimates independently,respectively. Then, we fix ρt and feed it into the model-basedoptimization module, which evaluates a lower bound y′t bysolving an approximate problem of (4). Two target values ytand y′t can be merged by simply adopting the larger one inthe following learning process. That is, for y′t > yt, we usey′t as the target value for the following DNN training andsimultaneously update the beamforming strategy (w′t,θ

′t) in

the action, i.e., at = (ρt,w′t,θ′t). We may also have y

′t < yt

when the learning becomes more stable. In this case, wefollow exactly the output of the actor network. The integrationof the model-based optimization and the model-free learningmay help the DDPG algorithm to adapt faster in the earlystage. Moreover, the optimization-driven target value y′t isindependent of the critic network. This implies that the targetvalue y′t can be more stable than the output yt of the targetQ-network during the training of online Q-network. Such adecoupling between the online Q-network and its target canreduce the performance fluctuation in training and thus it isexpected to stabilize the learning in a shorter time.

2) Model-based Approximation: Till this point, we aim atfinding an efficiently tractable approximation of problem (4) toderive the lower bound on yt. With a fixed ρ, we first determinea simple heuristic solution to the phase vector θ and thenfocus on the optimization of the AP’s transmit beamformingby reformulating a convex approximation of problem (4).The intuition behind our solution lies is that the IRS-assistedchannel ρHfθ can be aligned with the direct channel g for alarge-size IRS. That is, we can always find a phase vector θsuch that Hfθ = κg, where κ ∈ R+ is a scalar constant. Thisallows us to simplify the constraint in (4b).

Given the phase vector θ and the channel gain κ, we thenfocus on the optimization of the AP’s active beamforming inproblem (4), which is tightly coupled with the uncertain chan-

5

Experience replay memory

IRS-assiste

d W

irele

ss Netw

ork

Target Value Estimation

Model-based Optimization

(Lower bound)

Policy network(Actor)

Online Q-network

TD error

Transition sample(𝒔𝑡 , 𝒂𝑡 , 𝑟𝑡 , 𝒔𝑡+𝟏)

Merged value network (Critic)

action 𝒂𝑡

state st

(𝒔𝑡 , 𝑟𝑡)Training DDPG

Merged action set

(𝒘𝑡, 𝜽𝑡)(𝒘𝒕′ , 𝜽𝒕

′)

𝜌𝑡

Update DNN parameter 𝝎𝑡

Fig. 2: The optimization-driven DDPG framework for active andpassive beamforming in an IRS-assisted MISO downlink system.

nel matrices Hf and H. Fortunately, we have the followingequivalence to the worst-case constraints in (4b)-(4c).

Proposition 1: Given the IRS’s passive beamforming (ρ,θ),the constraints in (4b) and (4c) have the following equivalentreformations, respectively:[

ρ2(θθH ⊗W

)+ tIMN αρ(θ ⊗W)g

αρgH(θ ⊗W)H α2gHWg − γ1 − tδ2f

]� 0,

(12a)[Wc + τIMN , Wcvec(H̄)vec(H̄)HWc, γ̄0 − Nµη(1−ρ2) − τδ

2h

]� 0, (12b)

for some t ≥ 0 and τ ≥ 0, where α , (1 + ρκm), Wc ,IN ⊗W, and γ̄0 , vec(H̄)HWcvec(H̄). IMN is the identitymatrix with size MN . The semidefinite matrix W is a rank-one relaxation of wwH , i.e., W � wwH .

The detailed proof for Proposition 1 is relegated to ouronline technical report [21]. It is clear that with a fixed ρProposition 1 transforms the semi-infinite constraints in (4b)and (4c) into linear matrix inequalities in terms of W andthe non-negative auxiliary variables (t, τ). Till now, we caneasily derive a achievable lower bound on problem (4) by thefollowing convex problem:

minW�0,t≥0,τ≥0

{Tr(W) : (12a) and (12b)}, (13)

which can be solved efficiently by the interior-point al-gorithms. The linear beamforming w can be retrieved byeigenvalue decomposition if the matrix solution W to (13)is of rank one. Otherwise we can extract an approximaterank-one solution via Gaussian randomization [22]. Oncewe determine w, we can update the energy efficiency as(g + ρtH̄f ,tθt)

Hwt|2||wt||−2, which serves as the model-based evaluation of the target y′t.

IV. NUMERICAL RESULTS

In the simulation, we consider fixed network topology toverify the learning performance of the optimization-drivenDDPG algorithm. As shown in Fig. 1, the distances in metersare given by d0 = 1, d1 = 1, and d2 = 2. The signal

0 25000 50000 75000 100000 125000 150000 175000 200000Episode

400

500

600

700

800

900

AP's

trans

mit

powe

r

Optimization-driven DDPGModel-free DDPG

(a) AP’s transmit power

0 25000 50000 75000 100000 125000 150000 175000 200000Episode

0.70

0.75

0.80

0.85

0.90

--

the

mag

nitu

de o

f ref

lect

ion

Optimization-driven DDPGModel-free DDPG

(b) Magnitude of reflection

Fig. 3: Dynamics of the AP’s transmit power and the magnitudeof IRS’s reflection coefficients in two DDPG algorithms. The solidline denotes the median of 50 repetitions and the shaded regions indifferent colors cover 10th to 90th percentiles.

propagation satisfies a log-distance model with the path lossat the reference point given by L0 = 30 dB and the path lossexponent equal to 2. The energy harvesting efficiency is set asη = 0.8. In the following, we firstly demonstrate the learningperformance of the proposed algorithm and then study theimpact of different parameters on the AP’s transmit power.

Figure 3(a) demonstrates the dynamics of the AP’s min-imum transmit power during the learning of the proposedOptimization-driven DDPG algorithm. We also compare itto the conventional DDPG algorithm, denote as model-freeDDPG in Fig. 3. The common observation is that the AP’stransmit powers in both cases decrease gradually during thetraining process and converge eventually at two stable values.However, the optimization-driven DDPG converges faster thanthe model-free DDPG algorithm and also achieves a significantperformance improvement in terms of the AP’s transmit power.The reason is that the optimization-driven DDPG uses a better-informed estimation for the target value to guide its searchfor the optimal policy, which can achieve faster learning ratein the early stage of training compared to the model-freeDDPG that learns from scratch. Furthermore, in Fig. 3(b) werecord the magnitude ρ (the PS ratio) of the IRS’s reflectioncoefficient in two algorithms. It is clear that the PS ratio ρincreases gradually and converges to the same value within200k episodes. The comparison in Fig. 3(b) also reveals thatthe optimization-driven DDPG algorithm achieves a fasterconvergence rate compared to that of the model-free DDPGalgorithm. Besides, we can observe from Fig. 3 that thelearning curves are more stable in the proposed optimization-driven DDPG algorithm. The reason is that the model-basedtarget estimation y′t is independent or decomposed from theonline Q-network.

In the following, we evaluate the performance of the pro-posed optimization-driven DDPG against different parameters,including a) the receiver’s worst-case SNR requirement γ1 inproblem (4), b) the size N of the IRS’s scattering elements,and c) the channel’s uncertainty level β, defined by theratio between error estimate and the mean channel estimate,i.e., βh , δ2h/Tr(H̄H̄H) and βf , δ2f /Tr(H̄fH̄Hf ). In thesimulation, we set βh = βf = β. Fig. 4(a) shows that theAP’s minimum transmit power increases with the increasein the receiver’s SNR requirement γ1. For the same SNR

6

0.20 0.25 0.30The receiver's SNR requirement

350

400

450

500

550

600

AP's

trans

mit

powe

r

= 0.10 = 0.15 = 0.20

(a) Impact on the SNR requirement

20 50 80The size of IRS

200

250

300

350

400

450

500

AP's

trans

mit

powe

r

= 0.10 = 0.15 = 0.20

(b) Impact on the IRS’s size

Fig. 4: AP’s transmit power increases with the receiver’s SNRrequirement and decreases with the size of IRS’s scattering elements.

TABLE I: Comparison of running time between the SDR-basedoptimization and the optimization-driven DDPG algorithms

M ×N Running time (in milliseconds)SDR DRL40 6.22 12.0360 12.91 12.4180 19.27 12.82100 28.67 13.25200 136.20 15.19

requirement, the AP’s transmit power also increases with ahigher uncertainty level β. Such an increase in the AP’stransmit power can be viewed as the price of robustness toensure worst-case performance guarantee. In Fig. 4(b), weshow the dynamics of the AP’s transmit power with a differentnumber of the IRS’s scattering elements. Though a higheruncertainty level leads to increased transmit power, such anegative effect can be alleviated by using a larger-size IRS.

To show the scalability of the DRL approach, we com-pare the average running time of an SDR-based optimizationmethod to solve problem (4) and that of the optimization-driven DDPG algorithm. The results are shown in Table I. Notethat the SDR-based optimization method has a polynomiallyincreasing computational complexity in terms of the problemsize M × N . Hence, it becomes very time-consuming witha large number of the IRS’s scattering elements. On thecontrary, the optimization-driven DDPG algorithm achievesnearly O(1) complexity in the size M×N , as it only relies ona light-weight convex optimization to derive a lower bound formore efficient learning. Such a low complexity makes it verysuitable for practical deployment, especially with a large-sizeIRS and a large number of the AP’s transmit antennas.

V. CONCLUSIONS

In this paper, we propose a novel DRL approach to solvea robust power minimization problem in an IRS-assistedMISO system, subject to the receiver’s worst-case data raterequirement and the IRS’s worst-case power budget constraint.Different from the conventional DDPG algorithm, we devisethe optimization-driven DDPG algorithm that combines thebenefits of both model-free learning and model-based opti-mization. Our simulation results demonstrate that the newDDPG algorithm can guide the search for the joint activeand beamforming more efficiently. Both the learning rate and

reward performance can be improved significantly comparedto the conventional model-free DDPG algorithm.

REFERENCES[1] M. D. Renzo, M. Debbah, D. T. P. Huy, A. Zappone, M. Alouini,

C. Yuen, V. Sciancalepore, G. C. Alexandropoulos, J. Hoydis,H. Gacanin, J. de Rosny, A. Bounceu, G. Lerosey, and M. Fink, “Smartradio environments empowered by AI reconfigurable meta-surfaces: Anidea whose time has come,” EURASIP J. Wireless Commun. Network.,vol. 129, 2019.

[2] Q. Wu and R. Zhang, “Towards smart and reconfigurable environment:Intelligent reflecting surface aided wireless network,” IEEE Commun.Mag., vol. 58, no. 1, pp. 106–112, Jan. 2020.

[3] S. Gong, X. Lu, D. T. Hoang, D. Niyato, L. Shu, D. I. Kim, and Y.-C.Liang, “Towards smart wireless communications via intelligent reflectingsurfaces: A contemporary survey,” arXiv preprint arXiv:1912.07794,2019. [Online]. Available: https://arxiv.org/abs/1912.07794.

[4] W. Qingqing and Z. Rui, “Intelligent reflecting surface enhanced wire-less network: Joint active and passive beamforming design,” in proc.IEEE GLOBECOM, Dec. 2018.

[5] X. Yu, D. Xu, D. W. K. Ng, and R. Schober, “Power-efficient resourceallocation for multiuser MISO systems via intelligent reflectingsurfaces,” arXiv preprint arXiv:2005.06703, 2020. [Online]. Available:https://arxiv.org/abs/2005.06703.

[6] X. Yu, D. Xu, and R. Schober, “Enabling secure wireless communica-tions via intelligent reflecting surfaces,” in proc. IEEE GLOBECOM,Dec. 2019.

[7] B. Lyu, D. T. Hoang, S. Gong, and Z. Yang, “Intelligent reflectingsurface assisted wireless powered communication networks,” in proc.IEEE WCNC Workshops, Apr. 2020.

[8] T. Jiang and Y. Shi, “Over-the-air computation via intelligent reflectingsurfaces,” in proc. IEEE GLOBECOM, Dec. 2019.

[9] A. U. Makarfi, K. M. Rabie, O. Kaiwartya, K. Adhikari,X. Li, M. Quiroz-Castellanos, and R. Kharel, “Reconfigurableintelligent surfaces-enabled vehicular networks: A physical layersecurity perspective,” arXiv preprint arXiv:2004.11288, 2020. [Online].Available: https://arxiv.org/abs/2004.11288.

[10] C. Huang, G. C. Alexandropoulos, C. Yuen, and M. Debbah, “Indoorsignal focusing with deep learning designed reconfigurable intelligentsurfaces,” in proc. IEEE SPAWC, Jul. 2019.

[11] J. Gao, C. Zhong, X. Chen, H. Lin, and Z. Zhang, “Unsupervisedlearning for passive beamforming,” arXiv preprint arXiv:2001.02348,2020. [Online]. Available: https://arxiv.org/abs/2001.02348.

[12] A. Taha, M. Alrabeiah, and A. Alkhateeb, “Deep learning for largeintelligent surfaces in millimeter wave and massive MIMO systems,” inIEEE GLOBECOM, 2019, pp. 1–6.

[13] Y. Song, M. R. A. Khandaker, F. Tariq, and K.-K. Wong, “Trulyintelligent reflecting surface-aided secure communication using deeplearning,” arXiv preprint arXiv:2004.03056, 2020. [Online]. Available:https://arxiv.org/abs/2004.03056.

[14] K. Feng, Q. Wang, X. Li, and C. Wen, “Deep reinforcement learningbased intelligent reflecting surface optimization for MISO communica-tion systems,” IEEE Wireless Commun. Lett., vol. 9, no. 5, pp. 745–749,2020.

[15] H. Yang, Z. Xiong, J. Zhao, D. Niyato, L. Xiao, and Q. Wu, “Deepreinforcement learning based intelligent reflecting surface for securewireless communications,” arXiv preprint arXiv:2002.12271, 2020.[Online]. Available: https://arxiv.org/abs/2002.12271.

[16] A. Taha, Y. Zhang, F. B. Mismar, and A. Alkhateeb, “Deepreinforcement learning for intelligent reflecting surfaces: Towardsstandalone operation,” arXiv preprint arXiv:2002.11101, 2020. [Online].Available: https://arxiv.org/abs/2002.11101.

[17] C. Huang, A. Zappone, G. C. Alexandropoulos, M. Debbah, andC. Yuen, “Reconfigurable intelligent surfaces for energy efficiency inwireless communication,” IEEE Trans. Wireless Commun., vol. 18, no. 8,pp. 4157–4170, Aug. 2019.

[18] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction.MIT press, 2018.

[19] N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P. Wang, Y. C.Liang, and D. I. Kim, “Applications of deep reinforcement learning incommunications and networking: A survey,” IEEE Commun. Surv. Tut.,vol. 21, no. 4, pp. 3133–3174, Fourthquarter 2019.

[20] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa,D. Silver, and D. Wierstra, “Continuous control with deep reinforcementlearning,” in Proc. Int. Conf. Learning Representations (ICLR), San Juan,Puerto Rico, May 2016.

https://arxiv.org/abs/1912.07794.https://arxiv.org/abs/2005.06703.https://arxiv.org/abs/2004.11288.https://arxiv.org/abs/2001.02348.https://arxiv.org/abs/2004.03056.https://arxiv.org/abs/2002.12271.https://arxiv.org/abs/2002.11101.

7

[21] Y. Deng, Y. Zou, S. Gong, B. Lyu, D. T. Hoang, and D. Niyato, “Robustbeamforming for irs-assisted wireless communications under channeluncertainty,” 2020. [Online]. Available: https://arxiv.org/abs/2005.06754.

[22] Z.-Q. Luo, W. Ma, A. M. So, Y. Ye, and S. Zhang, “Semidefiniterelaxation of quadratic optimization problems,” IEEE Signal Process.Mag., vol. 27, no. 3, pp. 20–34, 2010.

https://arxiv.org/abs/2005.06754.

I IntroductionII System modelII-A SNR and Energy BudgetII-B Channel Uncertainty Model

III Robust Active and Passive BeamformingIII-A Deep Reinforcement Learning ApproachIII-A1 MDP ReformulationIII-A2 DDPG for Continuous Control

III-B Optimization-driven Learning StrategyIII-B1 Merge Model-free and Model-based EstimationsIII-B2 Model-based Approximation

IV Numerical ResultsV ConclusionsReferences