Web Service QoS Prediction Based on Adaptive Dynamic Programming Using Fuzzy Neural Networks for...

SPECIAL SECTION ON CHALLENGES FOR SMART WORLDS

Received September 3, 2015, accepted October 5, 2015, date of publication November 6, 2015, date of current versionNovember 19, 2015.

Digital Object Identifier 10.1109/ACCESS.2015.2498191

Web Service QoS Prediction Based on AdaptiveDynamic Programming Using Fuzzy NeuralNetworks for Cloud ServicesXIONG LUO, YIXUAN LV, RUIXING LI, AND YI CHENSchool of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, ChinaBeijing Key Laboratory of Knowledge Engineering for Materials Science, Beijing 100083, China

Corresponding author: X. Luo ([email protected])

This work was supported in part by the Fundamental Research Funds for the Central Universities, in part by the National Natural ScienceFoundation of China under Grant 61174103, Grant 61272357, and Grant 61300074, and in part by the National Key TechnologiesResearch and Development Program of China under Grant 2015BAK38B01.

ABSTRACT Recently, more and more traditional services are being migrated into a cloud computingenvironment that makes the quality of service (QoS) becomes an important factor for service selectionand optimal service composition while forming cross-cloud service applications. Considering the nonlinearand dynamic property of QoS data, it is so difficult to achieve dynamic prediction while designing a QoSprediction method with unsatisfactory prediction accuracy. It is thus desirable to explore how to design aneffective approach by incorporating some intelligent techniques into the QoS prediction method to improveprediction performance. In this paper, motivated by the adaptive critic design and Q-learning technique,we propose a novel QoS prediction approach to serve this purpose through the combination of fuzzyneural networks and adaptive dynamic programming (ADP), i.e., an online learning scheme. This approachextracts fuzzy rules from QoS data and employs the ADP method to parameter learning of the fuzzy rules.Moreover, we provide a convergence boundedness result for our proposed approach to guarantee the stability.Experimental results on a large-scale QoS service data set verify the prediction accuracy of our proposedapproach.

INDEX TERMS Quality of service (QoS), QoS prediction, fuzzy neural network, adaptive dynamicprogramming, cloud services.

I. INTRODUCTIONRecent years have seen the massive migration of traditionalservices to the cloud computing environment. These cloudservices on different cloud platforms are often invoked inweb services either deployed in the cloud by companies ordeployed in accordance with cloud applications. It is clearlycrucial to form cross-cloud service applications. There hasbeen a tremendous surge of interest on web services incloud environment [1]–[4]. Nowadays, these ever-increasingweb services could provide more functions and featuresin cloud environment to enhance the user experience [1].Within this trend, the performance of cloud services can beimproved when constructing the personalized cloud appli-cations through the integration of web services in cloudenvironment. Meanwhile, it poses a challenge in selectingqualifiedweb services for developing web service-based inte-gration cloud applications. The quality of service (QoS) is

becoming more and more crucial in describing web servicesduring the implementation of service selection and optimalservice composition. Generally, QoS is a set of nonfunctionalperformance indices of web services, including popularity,response time, failure probability, and many others [5], [6].Also, more and more attention has been paid to QoS whichhas been seen as a very hot topic in cloud computing [7], [8].Among those available performance indices, response timethat users can feel directly is one of the most important QoSproperties used to evaluate web services. The predicted QoSvalues have been widely used in service selection, serviceranking, and service composition under dynamic environ-ment [9]–[11]. Here, we discuss the prediction method forresponse time in QoS.

QoS is fundamental for cloud users, who expect providersto offer services of the advertised quality, and for serviceproviders, who need to find the right tradeoffs between

22602169-3536 2015 IEEE. Translations and content mining are permitted for academic research only.

Personal use is also permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

VOLUME 3, 2015

www.redpel.com +917620593389


X. Luo et al.: Web Service QoS Prediction Based on Adaptive Dynamic Programming ADP Using FNNs

QoS levels and operational costs [12]. The predictedresponse time is valuable for service composition, componentselection, task scheduling, and so on. At present, a numberof QoS prediction approaches for web services have beendeveloped. Generally speaking, the QoS predictionapproaches can be decomposed in two families. The first oneis the static prediction method. These methods (e.g. arith-metic average value method) are simple and easy toimplement, ignoring the individual factors of users, the ser-vice status, and the network environment. These predictionmethods do not reflect the nonlinear relationship of theQoS data [13], which may lead to a great error between thepredicted value and the actual value. The second one isthe dynamic prediction method, e.g. prediction via collabora-tive filtering, similarity measure, multi-dimensional weight-ing [14]–[19]. The main idea behind these methods is thatweb users who have similar historical QoS data on someservices, would have similar experiences on other services.But the difficulty in getting sufficient QoS data for similaritycalculation may lead to a large error under some cloudcomputing environment.

Considering the nonlinear and dynamic property ofQoS data as well as the lack of QoS data for calculationin some cases, we present a prediction approach based onadaptive dynamic programming (ADP) to avoid theabove limitations. ADP is a novel optimization tech-nique which combines concepts of reinforcement learning(e.g. Q-learning) and dynamic programming [20], [21].While dealing with complex systems, ADP uses a functionapproximation structure such as neural network (NN) toapproximate cost function, which avoids the curse of dimen-sionality and reduces the computation time. The approximateoptimal solution is obtained by using the offline iterationor the online update algorithms. In other words, ADP canlearn from a dynamic environment with less information. Theapplication of ADP is focused on nonlinear and complexsystems, such as aircraft system [22], power system [23],energy management system [24], and many others [25], [26].In view of this, due to its potential scalability of the adaptivecritic designs and the intuitiveness of Q-learning, ADP mayplay an important role in dealingwith the nonlinear predictionfor the QoS data.

Meanwhile, we propose to use fuzzy logic to express theinner relationship of the QoS data that users who have sim-ilar QoS data on some services would have similar expe-riences on other services. Fuzzy logic can deal with theQoS of web service in context-aware environment, whichcan be expressed with linguistic variables and membershipfunctions. Thus, fuzzy logic has been used in QoS man-agement system [27] and QoS prediction with fuzzy clus-tering [28]. But, on the other hand, fuzzy logic lacks thelearning ability and the adaptability while extracting fuzzyrules from QoS data. Then, it can be combined with NN toavoid such limitation. By combining fuzzy logic with NN,fuzzy neural network (FNN) provides a mathematical tool todeal with the nonlinear and dynamic property of QoS data

in context-aware environment. Specifically, FNN not onlyextracts fuzzy rules from the data to express the dynamicproperty of the system, but also simply adjusts the parametersof the network during the learning process to enhance theadaptability of the network [29]. Therefore, FNN shows greatadaptability and flexibility in developing a nonlinear modelfor QoS prediction.

In consideration of all the issues in QoS predictionmentioned above, we may conduct a QoS prediction thoughthe combination of FNN and ADP. In our previous researchwork of [30], by using FNN, the direct heuristic dynamic pro-gramming as a special ADP was presented in a longitudinalcontrol of hypersonic vehicles. Although the control systemframework was implemented, there is only a single actionoutput under its scheme. Moreover, there is no stability proofrelated to the convergence of proposed ADP approach, whichlimited its application. Motivated by it, the purpose of thispaper is to present a novel QoS prediction service schemewhich fuses ADP with FNN. Within the proposed scheme,FNN is employed to approximate the cost-to-go functionand the action network in ADP. In addition to novelQoS prediction scheme, we also provide a associatedLyapunov stability proof and its convergence result to guar-antee a uniform ultimate boundedness (UUB) property forthe weight errors of NN in the proposed FNN-based ADPscheme. These results are better than the previous one in [30].Therefore, due to its unique features of ADP as well asstrong adaptability and flexibility of FNN in context-awareenvironment, our proposed scheme may serve the purpose ofeffectively predicting those QoS data for cloud services.

The rest of this paper is organized as follows. Section 2presents the background of QoS prediction. Section 3describes the characteristics of FNN. Section 4 presentsthe architecture and the convergence analysis of theFNN-based ADP prediction algorithm. The QoS predictionexperiments using our proposed scheme are illustrated insection 5. Finally, section 6 provides a conclusion.

II. PROBLEM STATEMENTSince many web services have the similar functions withcloud applications, consumers are not possible to use everyweb service. Therefore, QoS data of unused web servicesplays an important role in providing suitable web services toconsumers.

TABLE 1. A user-component matrix.

Let us first consider an example in Table 1. Each entrypresents the response time of a web service invoked bya user, which is a particular QoS property value. Thenumeric entries in the table mean the response time for users

VOLUME 3, 2015 2261



X. Luo et al.: Web Service QoS Prediction Based on ADP Using FNNs

who have invoked the services. The blank entries correspondto the users who have not invoked the services.

For the purpose of service composition or service ranking,we need to know all the QoS data of web services related tousers. However, the real QoS data is similar to the example wepresent here. Therefore, it is significant to predict the blankentries before any QoS-based service selection or serviceranking. As a result, the problem we study in this paper ishow to precisely predict the blank entries in accordance withthe existing entries.

III. FUZZY NEURAL NETWORKWith the development of intelligent techniques, fuzzy logicand NN have made remarkable achievements in their respec-tive fields. With the improvement of the fuzzy theory and thenew upsurge of NN research in 1980s, many researchers paidmore attention to the fusion of those two fields. They utilizedthe superiority of fuzzy theory and NN, forming the conceptof FNN.

Adaptive-network-based fuzzy inference system (ANFIS)is a NN implementation of Takagi-Suguno (T-S) fuzzy infer-ence system [31]. ANFIS constructs a set of fuzzy if-thenrules automatically and optimizes the rules through the self-learning of NN. This avoids such limitation that fuzzy if-thenrules mainly come from the experience of experts. Comparedwith back propagate (BP) network, ANFIS has stronger self-learning ability, robustness, and adaptability while it canapproximate any nonlinear function.

FIGURE 1. ANFIS architecture.

Fig. 1 shows an architecture of ANFIS. The ANFIS con-sists of five layers. Here, we assume that ANFIS has n inputs(i.e., x1, · · · , xn) and 1 output (i.e., y). There areN fuzzy sets.Suppose that the rule base contains N fuzzy if-then rules ofT-S type:

Rule j : IF x1 = A1j and · · · and xn = Anj,

THEN wj =n∑i=1

zijxi + zn+1,j,

z1j, · · · , zn+1,j ∈ R; j = 1, 2, · · · ,N .

Here, Aij represents the fuzzy variable where i = 1, · · · , nand j = 1, · · · ,N .Layer 1 is the fuzzification layer. This layer defines the

membership function of the nodes. Normally the function isa Guassian function:

µAij (xi) = exp

{−(xi − cij)2

σ 2ij

}. (1)

where cij and σij are the excepted value and standard deviationof Gaussian function, respectively.

Layer 2 is the rule layer. Each node multiplies those inputsand generates the outputs as follows:

vj =n∏i=1

µAij (xi). (2)

The outputs of this layer stand for the firing strength of a rule.Layer 3 is the normalization layer. Nodes of layer 3 gener-

ate the weights according to:

vj =vjN∑j=1

vj

. (3)

Layer 4 is the defuzzification layer. Nodes in this layer arecalculated by:

vjwj = vj

(n∑i=1

zijxi + zn+1,j

). (4)

Layer 5 is the output layer. The node stands for the ANFISoutput y by calculating the sum of outputs of defuzzificationlayer.

y =N∑j=1

vjwj. (5)

Due to the combination of BP algorithm and least squareestimation (LSE) algorithm in its implementation, ANFISachieves good performance with fast learning speed. In thetraining process, the parameters of if-then rules can be iden-tified via LSE in the forward pass. In the backward pass, theerror rates propagate backward while the expectations andvariances of Guassian function are updated via BP algorithm.Therefore it is more accurate and efficient through the use ofthat hybrid algorithm.

IV. QoS PREDICTION SCHEMEIn the past decades, the ADP method was proposed throughthe use of NN to approximate the cost function. Accordingto the critic’s output, the ADP can be categorized as:heuristic dynamic programming (HDP), dual heuristic pro-gramming (DHP), and globalized dual heuristic program-ming (GDHP) [21], [32]. The action dependent version ofHDP and DHP are formed when the critic’s inputs areaugmented with the controller’s output. In this paper, ourapproach is proposed on the basis of HDP.

2262 VOLUME 3, 2015




FIGURE 2. ADP architecture.

A. ADP STRUCTUREAs shown in Fig. 2, this ADP is structured to estimatethe cost function J (t) in the Bellman equation of dynamicprogramming:

J (t) =∞∑

k=t+1

αk−t−1r(t), (6)

where α is a discount factor bounded within (0, 1) and r(t) isthe external reinforcement signal. In addition, x(t) is the stateof the plant and u(t) is the control signal of the plant attime t .

Through the use of the critic network of ADP, the J is cal-culated as an approximator of the optimal value function J∗.It is implemented by minimizing the following error measureover time:

Ec(t) =12× e2c(t), (7)

ec(t) = αJ (t)− (J (t − 1)− r(t)). (8)

Meanwhile, the action network of ADP is to backpropagatethe error between the desired objective Uc and the approxi-mator J∗. Since we have defined ‘‘0’’ as the reinforcementsignal for ‘‘success’’ in our proposed approach, Uc is set to 0all the time without loss of generality. Then, by minimizingthe following performance error measure, the weights in theaction network can be updated.

Ea(t) =12× e2a(t), (9)

ea(t) = J (t)− Uc(t). (10)

B. QoS PREDICTION BASED ON ADP USING ANFISOur proposed approach aims to predict the missing QoS valueof web service mentioned in Section 2. To deal with thenonlinear and dynamic properties of QoS data in context-aware environment, the ADP using FNN is proposed toenhance the adaptability and flexibility of Internet. Unlikethe traditional actor-critic design in ADP normally usingfeedforward networks with one hidden layer to implement thecritic network and the action network, our proposed approachemploys FNN to construct those two networks.

The online training and optimization algorithm in ADPwith ANFIS is shown in Algorithm 1. In traditional ADP,x(t) is the state of the plant and u(t) is the control signal of

Algorithm 1 The ADP Using ANFIS

1 initialize t = 0, x(0), and u(0);2 apply x(t) to the action network and obtain u(t);3 apply u(t) to the system model and obtain x(t + 1);4 apply x(t + 1) and u(t) to the critic network, and obtainJ (t);

5 update weights of the critic network till the requirementon Ec(t) is satisfied;

6 update weights of the action network till the requirementon Ea(t) is satisfied;

7 t = t + 1, continue from 2.

the plant at time t . In our proposed approach, x(t) stands fora vector of remaining QoS data used to predict a blank entryat time t . And u(t) is the predictive result of the blank entryat time t .

C. REDESCRIPTION OF ADP USING ANFISTo analyze the convergence of the ADP using ANFIS,we simplify the structure of ANFIS as illustratedin Figs. 3 and 4. Layer 1 and layer 2 in ANFIS are convertedto the input layer of NN. The output layer of NN consists ofthe layer 3, layer 4 and layer 5 in ANFIS.

In the simple ANFIS, pk and qk are the input and output ofthe hidden layer. The subscripts ‘‘a’’ and ‘‘c’’ mean the actionnetwork and critic network, respectively. In addition,c(t) and σ (t) stand for the expectation and variance ofGuassian function, respectively. And φ(·) represents a func-tion defined as:

φ(x) = exp{−12x}. (11)

FIGURE 3. Schematic diagram of ANFIS in critic network.

According to Fig. 3 and Fig. 4, the inputs of the criticnetwork include m states x(t) = (x1(t), · · · , xm(t))T andn actions u(t) = (u1(t), · · · , un(t))T . And wc(t) is the param-eter matrix of fuzzy rules in ANFIS. Applying the computingrule of exponential function to the combination of layer 1and layer 2 in ANFIS, the pc(t) can be written as follows.

VOLUME 3, 2015 2263




FIGURE 4. Schematic diagram of ANFIS in action network.

Let Nfc be the numbers of fuzzy rules in the critic network.The output J (t) can be expressed as:

J (t) =

Nfc∑k=1

wck (t)qck (t)

Nfc∑k=1

qck (t)

, (12)

qck (t) = φ(pck (t)), (13)

pck (t) =m∑i=1

(xi(t)− cci,k (t)

σci,k (t)

)2

+

n∑j=1

(uj(t)− ccj+m,k (t)

σcj+m,k (t)

)2

, k = 1, · · · ,Nfc

(14)

Unlike the traditional action network, the inputs includem states x(t) = (x1(t), · · · , xm(t))T . Let Nfa be the numbersof fuzzy rules in the action network. The output u(t) can beexpressed as:

uj(t) =

Nfa∑k=1

wajk (t)qak (t)

Nfa∑k=1

qak (t)

j = 1, · · · , n (15)

qak (t) = φ(pak (t)), (16)

pak (t) =m∑i=1

(xi(t)− cai,k (t)

σai,k (t)

)2

, k = 1, · · · ,Nfa (17)

D. CONVERGENCE ANALYSISIn this section, we refer to a UUB result in [33] and providea convergence bound for our proposed approach to show thatthe estimation errors of the network weights in this approachare UUB.

According to (13) and (16), φc(t) = (qc1 (t), · · · , qcNfc (t))T

and φa(t) = (qa1 (t), · · · , qaNfa (t))T are the hidden layer

neuron activation function vectors of the critic network andthe action network, respectively. The output layerweights wa(t) and wc(t) are initialized randomly and updated

during learning. Then, with (12) and (15), J (t) and u(t) canbe written with matrix forms:

J (t) = wTc (t)φc(t), (18)

u(t) = wTa (t)φa(t). (19)

The weight of critic network is updated as:

wTc (t + 1) = wc(t)− lc∂Ec(t)

∂ J (t)

∂ J (t)∂wc(t)

= wc(t)− αlcφc(t)

×[αwTc (t)φc(t)+ r(t)− wTc (t − 1)φc(t − 1)]T

(20)

where lc > 0 is the learning rate.The weight of action network is updated as:

wTa (t + 1) = wa(t)− la∂Ea(t)

∂ J (t)

∂ J (t)∂ u(t)

∂ u(t)∂wa(t)

= wa(t)− laφa(t)[wTc (t)C(t)][wTc (t)φc(t)]

T (21)

where la > 0 is the learning rate, and

C(t) =

(−φck (t)uj(t)− ccj+m,k (t)σ 2ccj+m,k

(t)

)Nfck=1

n

j=1

. (22)

Let w∗ be the optimal weight of network. Then we denote

the weight estimation error as w(t)def= w(t) − w∗. In the

following analysis, ‖ · ‖ represents the 2-norm.Definition 1: The solution w(t) is said to be uniformly

ultimately bounded (UUB) within a bound ε > 0, if for anyδ > 0 and t0 > 0, there exists a positive numberN = N (δ, ε),independent of t0, so that ‖ w(t) ‖6 ε for all t > N+ t0 when‖ w(t0) ‖< δ.Then, we will adopt a Lyapunov stability analysis frame-

work to prove the UUB property of our proposed approach.First, we present some assumptions as follows.Assumption 1: Let w∗c and w∗a be the optimal weights for

the critic and action network, respectively. They are bounded.Then,

‖ w∗c ‖6 wcm, ‖ w∗a ‖6 wam, (23)

where w∗c and w∗a are two positive constants.

Theorem 1: Let Assumption 1 hold. Take the reinforce-ment signal as r(t) ∈ [0, 1]. Let the critic and actionnetwork settings be (18) and (19), respectively. Let the weightupdate for the critic and action network be (20) and (21),respectively. Then the errors of network weights,wc(t) and wa(t), are UUB, provided the following conditionshold:√23< α<1, 0 < lc <

1α2 ‖φc(t)‖2

, 0 < la <1

‖φa(t)‖2,

(24)

Proof: See Appendix A.

2264 VOLUME 3, 2015




Theorem 2: Let Assumption 1 hold. Then, the errors ofnetwork weights, wc(t) and wa(t), are UUB, provided thefollowing conditions hold:√23< α < 1, 0 < lc <

1α2Nfc

, 0 < la <1Nfa

. (25)

where Nfc and Nfa are the numbers of fuzzy rules in the criticand action network, respectively.

Proof: See Appendix B.While using our approach, Theorem 2 gives a simple

sufficient condition for selecting learning rates in the pro-posed ADP architecture to maintain stability of the weightupdates.

V. EXPERIMENTSA. DATA SET DESCRIPTIONIn our experiments, the data set comes from theWS-DREAM [34], [35] including the QoS data of 5, 825real-world web services from 73 countries. The QoS valueis observed by 339 distributed computers and each computerinvokes all the 5, 825 web services by sending null operatingrequests. In this paper, we use a 339× 5825 user-componentmatrix which represents response time values for experiment.The statistics of the web service QoS response time data setis summarized in Table 2.

TABLE 2. The statistics of data set.

B. METRICSWe calculate mean absolute error (MAE) and mean absolutepercentage error (MAPE) to compare the prediction qualityof our proposed approach with other methods.

The MAE and MAPE are defined as follows:

MAE =1P

∑i,j

∣∣νij − νij∣∣ , (26)

MAPE =1P

∑i,j

∣∣∣∣ νij − νijνij

∣∣∣∣ , (27)

where νij is the actual QoS value of web service item jinvoked by service user i, νij is the prediction valueof each method, and P is the number of the predictedQoS values.

C. PERFORMANCE COMPARISONIn this section, the above metrics are compared betweenour proposed approach and the following popular predictionmethods.

TABLE 3. Performance comparison.

FIGURE 5. MAE of response time prediction.

FIGURE 6. MAPE of response time prediction.

• Arithmetic average value method (AVG). This methodis defined as

average =

N∑i=1νij

Nexist(28)

VOLUME 3, 2015 2265




FIGURE 7. Experimental result with 70% matrix density.

where νij is the actual QoS response time value that theuser i invokes the web service j, N is the number of allusers, Nexist is the number of the remaining users in thematrix who invoke web service j.

• User-based collaborative filtering method using Pearsoncorrelation coefficient (UPCC). This method predictsthe missing QoS data of users based on the collectedinformation from similar users [15], [17].

• Item-based collaborative filtering method using Pearsoncorrelation coefficient (IPCC). This method predicts themissing QoS data of users based on the collected infor-mation from similar service items [14].

• UIPCC. This method combines UPCC with IPCC formissing QoS data prediction [16].

In our experiments, the number of the fuzzy rules in boththe critic and the action networks are set to Nfc = Nfa = 3.According to Theorem 2, the discount factor is set to

α = 0.95 ∈ (√23 , 1), and the learning rates of the critic and

the action networks should satisfy lc < 10.952×3

≈ 0.369 andla < 1

3 ≈ 0.333, respectively. Then we choose the learningrate as lc = la = 0.3. In our experiment, we find that thiscommon fixed learning rate will result in slow convergencerate of ADP. So we use the learning rates lc = la = 0.3 inthe initial training phase of NN. And they should decreaseto a value during the learning process. Here, the learningrates are decreased by 0.05 every 50 times until theyreach 0.005.

In order to compare the performance of each approach,we set different sparsity to randomly remove some entriesfrom the matrix. For instance, 10% means that 10% entriesare removed randomly, and the remaining entries are used topredict the missing entries.

The experimental results are show in Table 3. From thistable, we can find that our proposed approach has betterperformance (smaller MAE and MAPE values) than othermethods with different matrix densities, especially with highmatrix density. Figs. 5 and 6 show the experimental resultsof response time prediction. From those figures, we concludethat the performance of our proposed approach is stable asthe performance of other methods becomes worse with theincrease of the matrix density. An experimental result with70% matrix density is shown in Fig. 7, which shows thedifferences of the performance in reflecting the trend ofQoS data among these methods. Due to its potential scal-ability of the adaptive critic designs and the intuitivenessof Q-learning in ADP as well as strong adaptability andflexibility of FNN in context-aware environment, our pro-posed scheme has its unique features in dealing with thoseQoS data. Compared with other methods, the improvementof our proposed approach verifies that the idea involvedin the fusion of ADP and FNN for QoS prediction iseffective.

VI. CONCLUSIONQoS evaluates the web services while playing an importantrole in web service selection and service recommendationfor cloud applications. In this paper, we present a novelQoS prediction scheme through the fusion of FNN and ADP.Considering the nonlinear and dynamic property of QoS data,we construct FNN in the critic as well as the action networkunder ADP architecture, and develop the weight updatingmethod to adjust the parameters of the networks with thepurpose of optimizing a set of fuzzy if-then rules. In addition,we provide the stability analysis for our proposed approachto guarantee the convergence of scheme. These conver-gence results are better than the previous one, which providea guideline for the selection of parameters used in ourFNN-based ADP scheme. Experiments are conducted ona large-scale real-world QoS service data set to predictthe response time in QoS. Compared with other traditionalmethods, the experimental results shows the proposedapproach improves the accuracy of prediction for responsetime in QoS.

2266 VOLUME 3, 2015




APPENDIX APROOF OF THEOREM 1Let ζc(t) and ζa(t) be the approximation errors of the criticand the action network outputs, respectively. Then,

ζc(t) = (wc(t)− w∗c )Tφc(t) = wT

c (t)φc(t), (A.29)ζa(t) = (wa(t)− w∗a)

Tφa(t) = wTa (t)φa(t). (A.30)

Select the Lyapunov function as:

V (t) = V1(t)+ V2(t)+ V3(t), (A.31)

where

V1(t) =1lctr[wTc (t)wc(t)

], (A.32)

V2(t) =1γ la

tr[wTa (t)wa(t)

], (A.33)

V3(t) =29‖ζc(t − 1)‖2. (A.34)

where γ > 0 is a weighting factor.The first difference of V1(t) is given by:

4V1(t) = −α2‖ζc(t)‖2 − α2(1− lcα2‖φc(t)‖2

)×

∥∥∥ζc(t)+ (w∗c )Tφc(t)+

1αr(t)

−1αwTc (t − 1)φc(t − 1)

∥∥∥2+∥∥α(w∗c )Tφc(t)+ r(t)− wTc (t − 1)φc(t − 1)

∥∥2.(A.35)

By applying the Cauchy-Schwarz inequality for (A.35),we have

4V1(t) 6 −α2‖ζc(t)‖2 − α2(1− lcα2‖φc(t)‖2

)×

∥∥∥ζc(t)+ (w∗c )Tφc(t)+

1αr(t)

−1αwTc (t − 1)φc(t − 1)

∥∥∥2+ 2

∥∥∥α(w∗c )Tφc(t)+ r(t)− 23wTc (t − 1)φc(t − 1)

−13(w∗c )

Tφc(t − 1)∥∥∥2

+29‖ζc(t − 1)‖2. (A.36)

The first difference of V2(t) is given by:

4V2(t) =1γ

[−

[‖wTc (t)C(t)‖

2− la‖wTc (t)C(t)‖

2‖φa(t)‖2

]× ‖wTc (t)φc(t)‖

2− ‖wTc (t)C(t)‖

2‖ζa(t)‖2

+ ‖wTc (t)φc(t)− wTc (t)C(t)ζa(t)‖

2]. (A.37)

By applying the Cauchy-Schwarz inequality for (A.37), wehave

4V2(t) 61γ

[−

(1− la‖φa(t)‖2

)‖wTc (t)C(t)‖

2‖wTc φc(t)‖

2

+ 4‖(w∗c )Tφc(t)‖2 + ‖wTc (t)C(t)‖

2‖ζa(t)‖2

+ 4‖ζc(t)‖2]. (A.38)

The first difference of V3(t) is given by:

4V3(t) =29

(‖ζc(t)‖2 − ‖ζc(t − 1)‖2

). (A.39)

Substituting (A.36), (A.38), and (A.39) into the first differ-ence of V (t), we have

4V (t) = 4V1(t)+4V2(t)+4V3(t)

6−

(α2 −

29−

4γ

)‖ζc(t)‖2 − α2

(1−lcα2‖φc(t)‖2

)×

∥∥∥ζc(t)+ (w∗c )Tφc(t)+

1αr(t)

−1αwTc (t − 1)φc(t − 1)

∥∥∥2−1γ(1− la‖φa(t)‖2)‖wTc (t)C(t)‖

2‖wTc (t)φc(t)‖

2

+Z2(t), (A.40)

where

Z2(t) = 2∥∥∥α(w∗c )Tφc(t)+ r(t)− 2

3wTc (t − 1)φc(t − 1)

−13(w∗c )

Tφc(t − 1)∥∥∥2 + 1

γ‖wTc (t)C(t)‖

2‖ζa(t)‖2

+4γ‖(w∗c )

Tφc(t)‖2. (A.41)

Applying the Cauchy-Schwarz inequality for (A.41) andusing Assumption 1, we have

Z2(t) 6 8(α2‖(w∗c )

Tφc(t)‖2 + r2(t)

+49‖wTc (t − 1)φc(t − 1)‖2 +

19‖(w∗c )

Tφc(t − 1)‖2)

+2γ‖wTc (t)C(t)‖

2(‖wTa (t)φa(t)‖

2+‖(w∗a)

Tφa(t)‖2)

+4γ‖(w∗c )

Tφc(t)‖2

6

(8α2 +

409+

4γ

)w2cmφ

2cm

+4γw2cmC

2mw

2amφ

2am + 8r2m = Z2

m, (A.42)

where wcm, wam, φcm, φam, Cm, and rm are the upper boundsof w∗c , w

∗a, φc(t), φa(t), C(t), and r(t), respectively.

According to (A.40), choose√23< α < 1, 0 < lc <

1α2 ‖φc(t)‖2

, 0 < la <1

‖φa(t)‖2,

(A.43)

and select γ satisfying

γ >4

α2 − 29

, (A.44)

then for any

‖ζc(t)‖ >Zm√

α2 − 29 −

4γ

, (A.45)

the first difference 4V (t) 6 0 holds.Using the Lyapunov theorem, this implies that the

NN weight estimation errors wc(t) and wa(t) are UUB. �

VOLUME 3, 2015 2267




APPENDIX BPROOF OF THEOREM 2According to the definition of function in (11), eachelement of activation function vectors φc(t) and φa(t),i.e., qc1 (t), · · · , qcNfc (t), qa1 (t), · · · , qaNfa (t), is boundedwithin (0, 1]. Then, we have

0 < lc <1

α2Nfc6

1

α2∑Nfc

k=1(qck (t))2=

1α2‖φc(t)‖2

,

(B.46)

0 < la <1Nfa6

1∑Nfak=1(qak (t))

2=

1‖φa(t)‖2

. (B.47)

According to the judging conditions (24) in Theorem 1,the estimate errors of network weights wc(t) and wa(t)are UUB. �

REFERENCES[1] T. H. Noor, Q. Z. Sheng, A. H. H. Ngu, and S. Dustdar, ‘‘Analysis of Web-

scale cloud services,’’ IEEE Internet Comput., vol. 18, no. 4, pp. 55–61,Jul./Aug. 2014.

[2] V. Chang, ‘‘The business intelligence as a service in the cloud,’’ FutureGenerat. Comput. Syst., vol. 37, pp. 512–534, Jul. 2014.

[3] C. Gravier, J. Subercaze, A. Najjar, F. Laforest, X. Serpaggi, andO. Boissier, ‘‘Context awareness as a service for cloud resource optimiza-tion,’’ IEEE Internet Comput., vol. 19, no. 1, pp. 28–34, Jan./Feb. 2015.

[4] R. Ranjan, J. Kolodziej, L. Wang, and A. Y. Zomaya, ‘‘Cross-layer cloudresource configuration selection in the big data era,’’ IEEE Cloud Comput.,vol. 2, no. 3, pp. 16–22, May/Jun. 2015.

[5] D. A. Menascé, ‘‘QoS issues in Web services,’’ IEEE Trans. InternetComput., vol. 6, no. 6, pp. 72–75, Nov./Dec. 2002.

[6] D. A. Menascé, ‘‘Composing Web services: A QoS view,’’ IEEE Trans.Internet Comput., vol. 8, no. 6, pp. 88–90, Nov./Dec. 2004.

[7] S. Misra, P. V. Krishna, K. Kalaiselvan, V. Saritha, and M. S. Obaidat,‘‘Learning automata-based QoS framework for cloud IaaS,’’ IEEE Trans.Netw. Serv. Manage., vol. 11, no. 1, pp. 15–24, Mar. 2014.

[8] C. Mao, J. Chen, D. Towey, J. Chen, and X. Xie, ‘‘Search-based QoS rank-ing prediction for Web services in cloud environments,’’ Future Generat.Comput. Syst., vol. 50, pp. 111–126, May 2014.

[9] H. Zheng, W. Zhao, J. Yang, and A. Bouguettaya, ‘‘QoS analysis forWeb service compositions with complex structures,’’ IEEE Trans. ServicesComput., vol. 6, no. 3, pp. 373–386, Jul./Sep. 2013.

[10] X. Chen, Z. Zheng, Q. Yu, and M. R. Lyu, ‘‘Web service recommenda-tion via exploiting location and QoS information,’’ IEEE Trans. ParallelDistrib. Syst., vol. 25, no. 7, pp. 1913–1924, Jul. 2014.

[11] S.-Y. Hwang, C.-C. Hsu, and C.-H. Lee, ‘‘Service selection for Webservices with probabilistic QoS,’’ IEEE Trans. Services Comput., vol. 8,no. 3, pp. 467–480, May/Jun. 2015.

[12] D. Ardagna, G. Casale,M. Ciavotta, J. F. Pérez, andW.Wang, ‘‘Quality-of-service in cloud computing: Modeling techniques and their applications,’’J. Internet Services Appl., vol. 5, no. 1, p. 11, 2014.

[13] A. Metzger, C.-H. Chi, Y. Engel, and A. Marconi, ‘‘Research challengeson online service quality prediction for proactive adaptation,’’ in Proc.Workshop Eur. Softw. Services Syst. Res.-Results Challenges (S-Cube),Zürich, Switzerland, 2012, pp. 51–57.

[14] P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl,‘‘GroupLens: An open architecture for collaborative filtering of netnews,’’in Proc. ACM Conf. Comput. Supported Cooperat. Work, New York, NY,USA, 1994, pp. 175–186.

[15] J. S. Breese, D. Heckerman, and C. Kadie, ‘‘Empirical analysis of predic-tive algorithms for collaborative filtering,’’ in Proc. 14th Conf. UncertaintyArtif. Intell., San Francisco, CA, USA, 1998, pp. 43–52.

[16] H. Ma, I. King, and M. R. Lyu, ‘‘Effective missing data prediction forcollaborative filtering,’’ in Proc. 30th Annu. Int. ACM SIGIR Conf. Res.Develop. Inf. Retr., New York, NY, USA, 2007, pp. 39–46.

[17] L. Shao, J. Zhang, Y. Wei, J. Zhao, B. Xie, and H. Mei, ‘‘Personalized QoSprediction for Web services via collaborative filtering,’’ in Proc. IEEE Int.Conf. Web Services, Salt Lake City, UT, USA, Jul. 2007, pp. 439–446.

[18] Z. Zheng, H. Ma, M. R. Lyu, and I. King, ‘‘Collaborative Web service QoSprediction via neighborhood integrated matrix factorization,’’ IEEE Trans.Services Comput., vol. 6, no. 3, pp. 289–299, Jul./Sep. 2013.

[19] L. Qi, J. Ni, X. Xia, H. Wang, and C. Yan, ‘‘A multi-dimensional weight-ing method for historical records in cloud service evaluation,’’ in Proc.IEEE 4th Int. Conf. Big Data Cloud Comput., Sydney, NSW, Australia,Dec. 2014, pp. 265–266.

[20] J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, ‘‘Adaptive dynamicprogramming,’’ IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 32,no. 2, pp. 140–153, May 2002.

[21] J. Si, A. G. Barto, W. B. Powell, and D. Wunsch, Eds., Handbook ofLearning and Approximate Dynamic Programming. Hoboken, NJ, USA:Wiley, 2004.

[22] S. N. Balakrishnan and V. Biega, ‘‘Adaptive-critic-based neural networksfor aircraft optimal control’’ J. Guid., Control, Dyn., vol. 19, no. 4,pp. 893–898, Jul./Aug. 1996.

[23] Y. Tang, H. He, J. Wen, and J. Liu, ‘‘Power system stability control for awind farm based on adaptive dynamic programming,’’ IEEE Trans. SmartGrid, vol. 6, no. 1, pp. 166–177, Jan. 2015.

[24] Q. Wei, D. Liu, G. Shi, and Y. Liu, ‘‘Multibattery optimal coordinationcontrol for home energy management systems via distributed iterativeadaptive dynamic programming,’’ IEEE Trans. Ind. Electron., vol. 62,no. 7, pp. 4203–4214, Jul. 2015.

[25] M. Fairbank, D. Prokhorov, and E. Alonso, ‘‘Clipping in neurocontrol byadaptive dynamic programming,’’ IEEE Trans. Neural Netw. Learn. Syst.,vol. 25, no. 10, pp. 1909–1920, Oct. 2014.

[26] J. Wang and R. Y. K. Fung, ‘‘Adaptive dynamic programming algorithmsfor sequential appointment scheduling with patient preferences,’’ Artif.Intell. Med., vol. 63, no. 1, pp. 33–40, Jan. 2015.

[27] F. Xia, W. Zhao, Y. Sun, and Y.-C. Tian, ‘‘Fuzzy logic control based QoSmanagement in wireless sensor/actuator networks,’’ Sensors, vol. 7, no. 12,pp. 3179–3191, 2007.

[28] M. Zhang, X. Liu, R. Zhang, and H. Sun, ‘‘A Web service recommenda-tion approach based on QoS prediction using fuzzy clustering,’’ in Proc.IEEE 9th Int. Conf. Services Comput., Honolulu, HI, USA, Jun. 2012,pp. 138–145.

[29] S.-I. Horikawa, T. Furuhashi, and Y. Uchikawa, ‘‘On fuzzy modeling usingfuzzy neural networks with the back-propagation algorithm,’’ IEEE Trans.Neural Netw., vol. 3, no. 5, pp. 801–806, Sep. 1992.

[30] X. Luo, Y. Chen, J. Si, and F. Liu, ‘‘Longitudinal control ofhypersonic vehicles based on direct heuristic dynamic programmingusingANFIS,’’ inProc. Int. Joint Conf. Neural Netw., Beijing, China, 2014,pp. 3685–3692.

[31] J.-S. R. Jang, ‘‘ANFIS: Adaptive-network-based fuzzy inference sys-tem,’’ IEEE Trans. Syst., Man, Cybern., vol. 23, no. 3, pp. 665–685,May/Jun. 1993.

[32] D. V. Prokhorov and D. C.Wunsch, ‘‘Adaptive critic designs,’’ IEEE Trans.Neural Netw., vol. 8, no. 5, pp. 997–1007, Sep. 1997.

[33] F. Liu, J. Sun, J. Si, W. Guo, and S. Mei, ‘‘A boundedness resultfor the direct heuristic dynamic programming,’’ Neural Netw., vol. 32,pp. 229–235, Aug. 2012.

[34] Z. Zheng, Y. Zhang, and M. R. Lyu, ‘‘Distributed QoS evaluation for real-world Web services,’’ in Proc. IEEE Int. Conf. Web Services, Miami, FL,USA, Jul. 2010, pp. 83–90.

[35] Y. Zhang, Z. Zheng, andM. R. Lyu, ‘‘Exploring latent features formemory-based QoS prediction in cloud computing,’’ in Proc. 30th IEEE Symp. Rel.Distrib. Syst., Madrid, Spain, Oct. 2011, pp. 1–10.

XIONG LUO received the Ph.D. degree fromCentral South University, China, in 2004. From2012 to 2013, he was with the School of Electrical,Computer and Energy Engineering, Arizona StateUniversity, USA, as a Visiting Scholar. He is cur-rently an Associate Professor with the Departmentof Computer Science and Technology, School ofComputer and Communication Engineering, Uni-versity of Science and Technology Beijing, China.His current research interests include machine

learning, cloud computing, cyber-physical systems, and computationalintelligence.

2268 VOLUME 3, 2015




YIXUAN LV is currently pursuing the master’sdegree with the Department of Computer Scienceand Technology, School of Computer andCommu-nication Engineering, University of Science andTechnology Beijing, China. His research inter-ests include cyber-physical systems and adaptivedynamic programming.

RUIXING LI is currently pursuing the master’sdegree with the Department of Computer Scienceand Technology, School of Computer andCommu-nication Engineering, University of Science andTechnology Beijing, China. Her research interestsinclude cloud computing and computationalintelligence.

YI CHEN received the master’s degree from theUniversity of Science and Technology Beijing,China, in 2014. Her research interests includefuzzy learning and cyber-physical systems.

VOLUME 3, 2015 2269



Web Service QoS Prediction Based on Adaptive Dynamic Programming Using Fuzzy Neural Networks for...

Education

Transcript of Web Service QoS Prediction Based on Adaptive Dynamic Programming Using Fuzzy Neural Networks for...