SD-LSSVR-based Decomposition-and-Ensemble Methodology with Application to Hydropower Consumption Forecasting

Shuai Wang , Ling Tang Institute of Policy and Management, Chinese Academy

of Sciences Beijing 100190, China

Graduate University of Chinese Academy of Sciences, Beijing 100049, China

e-mail:shuaiwangcas@gmail.com

Lean Yu Institute of Systems Science, Academy of Mathematics

and Systems Science, Chinese Academy of Sciences Beijing 100190, China

e-mail: yulean@amss.ac.cn

Abstract—Due to the distinct seasonal characteristics of hydropower, this study tries to propose a seasonal decomposition (SD) based least squares support vector regression (LSSVR) ensemble learning model for hydropower consumption forecasting. In the SD-LSSVR-based decomposition and ensemble model, the original hydropower consumption series are first decomposed into trend cycle, seasonal factor and irregular component. Then the LSSVR is used to predict the three different components independently. Finally, these prediction results of the three components are combined with another LSSVR to formulate an ensemble result as the final prediction. Experimental results reveal that the proposed novel method is very promising for time series forecasting with seasonality and nonlinearity for that it outperforms all the other benchmark methods listed in our study in both level accuracy and directional accuracy.

Keywords- Hydropower consumption forecasting; Seasonal decomposition; LSSVR ensemble learning

I. INTRODUCTION Hydropower, as a clean and renewable energy has been

playing an increasingly important role in the world energy system due to its technical, economical and environmental benefits. In addition, China has the richest hydraulic resource and a low development rate of hydropower (Which is only 27.3%)[1].Therefore, considering the inherent benefits of hydropower and the superiority of resources endowment in China, developing hydropower is of great importance to alleviate the current energy crisis and environmental pollution resulting from the rapid economic growth of China.

According to the “tenth-” and “eleventh-”five-year plans, Chinese government has proposed objectives of hydroelectric development and adjusted these objectives dynamically based on the practical development status. Therefore, an accurate prediction of hydropower consumption is important for policy makers to draw up hydropower development programs and modify the policy to ensure the expected objectives. Furthermore, a good hydropower consumption prediction is useful for the hydropower industry to optimize hydropower operation scheduling.

Current hydropower related forecasting studies are usually about the river flow discharges forecasting in hydropower station (e.g. Cheng, 2005[2]; Lima et al, 2009[3])

and the installation capacity forecasting of small hydropower (e.g. Zhou et al, 2009[4]). Only few researches on energy demand forecasting involve hydropower (Ediger and Akar, 2007[5]). The main difficulty of hydropower modeling comes from its seasonality and nonlinearity which the hydrological system exhibits [6].

It is obvious that the single prediction method has its own limits and disadvantages. For instance, traditional statistical and econometric techniques are used under linearity condition. Some nonlinear and artificial intelligence (AI) models have their own shortcomings, e.g. local minima and over fitting in artificial neural networks (ANN) models and sensitiveness to parameter selection in both support vector regression (SVR) and ANN models. Therefore, to remedy the above shortcomings, some hybrid methods have been used [7].

Due to the seasonal nature and nonlinear characteristic of hydropower series and motivated by hybrid methodology, we try to employ the “decomposition-and-ensemble” principle (or “divide and conquer” strategy), introduced by Wang et al. (2005) [8] and Yu et al. (2008) [9], to overcome the difficulty in forecasting hydropower consumption. Firstly, the original hydropower consumption series are decomposed into trend cycle, seasonal factor and irregular component. Then the LSSVR is used to predict the three different components independently. Finally, these prediction results of the three components are combined with another LSSVR to formulate an ensemble result as the final prediction.

This study aims to propose the SD-LSSVR based Decomposition-and-Ensemble methodology, and apply it to forecast the Chinese hydropower consumption. The rest of the article is organized as follows. Section II formulates the methodology. The experimental study is illustrated in section III. Section IV concludes the article.

II. METHODOLOGY FORMULATION In this section, the overall process of formulating the SD-

LSSVR based Decomposition-and-Ensemble methodology is presented. First, the SD technique and LSSVR algorithm are briefly introduced, and then the SD-LSSVR based decomposition-and-ensemble methodology is proposed.

2011 Fourth International Joint Conference on Computational Sciences and Optimization

978-0-7695-4335-2/11 $26.00 © 2011 IEEE

DOI 10.1109/CSO.2011.303

603

A. Seasonal decomposition (SD) approach In the existing seasonal decomposition methods, the

most commonly used method is the X-12-ARIMA method, which is the Census Bureau's latest seasonal adjustment program (U.S. Census Bureau, 2009[10]). Basically, X-12-ARIMA method decomposes the time series tx into three

components, namely trend-cycle component ttc , seasonal

factor tsf and irregular component tir . In X-12-ARIMA method, the additive and multiplicative forms are usually used to combine these components into the original data. They are defined below respectively:

t t t tx tc sf ir= + + (1)

t t t tx tc sf ir= × × (2) Comparing the two forms of seasonal decomposition, the

multiplicative decomposition is more suitable for most seasonal time series. Because of that the seasonal factor of the multiplicative form is a relative value of the original series and most seasonal time series with positive values has the characteristic that the scale of seasonal oscillations increases following the level of original series (Findley, 1998). So this study will employ the multiplicative form for seasonal decomposition and it is shortened as SD (M).

For comparison purpose, the ratio-to-moving-average (RMA) method which was first proposed by Macaulay (1931) is selected as the benchmark method relative to SD (M) method in this study[11]. Similarly, multiplicative forms of RMA is chosen for this study and it is shortened as RMA (M)

B. Least Squares Support Vector Regression (LSSVR) Support vector machine (SVM), which was firstly

proposed by Vapnik in 1995 is based on the principle of structural risk minimization. It has been proved to possess its own excellent capabilities in classification or prediction even for small sample by minimizing an upper bound of the generalization error [12]. Nevertheless, SVM training is a time consuming process when analyzing huge data. For this purpose, least squares support vector machine (LSSVM) is proposed to overcome these shortcomings [13]. Generally speaking, LSSVM can be categorized into LSSVR and LSSVC respectively for regression and classification purposes. In this study, the LSSVR is used for a useful tool for forecasting.

In LSSVR, the basic idea is first to map the original data x into a high-dimensional feature space via a nonlinear mapping function �(•) and then to make linear regression in this high-dimension feature space. Usually, the LSSVR regression function can be formulated as follows,

( ) ( )Ty x w x bϕ= + (3) where �(x) is called the nonlinear function mapping from

input space x into a high-dimensional feature space, and y is the estimated value. Coefficients wT and b are obtained by minimizing the upper bound of generalization error.

Accordingly, Eq. (3) can be transformed into the following optimization problem:

2

1

1 1min

2 2

. . ( ) , ( 1, 2, ..., )

lT

ii

T

i i i

w w

s t y w x b i l

γ ξ

ϕ ξ=

+

= + + =

� (4)

where iξ are the slack variables and γ is the penalty parameter. By introducing Lagrangian function and the KKT conditions for optimality, the final solution of the primal problem can be represented in the following form:

1

( ) ( , )l

i ii

f x w K x x b=

= +� (5)

In Eq. (5), K(•) is the so-called kernel function which can simplify the use of a mapping. In this study, Gaussian RBF kernel function ( )2( , ) exp 2i iK x x x x σ= − with a width of � is used.

C. SD-LSSVR based Decomposition-and-Ensemble method Depending on the techniques and methods presented in

the previous subsections, the SD-LSSVR based decomposition-and-ensemble method can be formulated, as illustrated in Fig. 1.

Figure 1. Overall process of the SD-LSSVR based decomposition-and-

ensemble methodology.

As shown in Fig.1, the proposed SD-LSSVR based decomposition-and-ensemble method is generally composed of the following three main steps:

(1) The original time series tx (t=1,2,…,T)is decomposed

into seasonal factor (SF) tsf , trend cycle (TC) ttc , and

irregular component (IR) tir via SD (M).

(2)For the decomposed components, tsf , ttc , and tir , the LSSVR is used as a single forecasting tool to model each decomposed component..

(3)The prediction results of SF, TC and IR are combined as an aggregated output as the final prediction result, using another LSSVR model.

To summarize, the proposed SD-LSSVR based decomposition-and-ensemble method can be abbreviated as

604

“SD (M) (decomposition)-LSSVR (single prediction)-LSSVR (ensemble prediction)” approach.

III. EXPERIMENTAL STUDY In this section, the data and evaluation criteria used in

this study is first described. Then, experimental results and corresponding analysis and explanations are reported.

A. Data Descriptions and Experiment Design In this study, the hydropower consumption data in China

are obtained from Wind Database (http://www.wind.com.cn/) and originally provided by National Bureau of Statistics of China. The sample data are monthly data of hydropower consumption in China, covering the period from January 1986 to February 2010 with a total of 290 observations, as shown in Fig. 2.

Jan86 Jan90 Jan95 Jan00 Jan05 Feb100

100

200

300

400

500

600

700

Hyd

ropo

wer

con

sum

ptio

n (H

undr

ed m

illio

n ki

low

att h

our)

Figure 2. The original monthly hydropower consumption series in China (Period: 01/1986-02/2010)

In these hydropower consumption data, the data from January 1986 to December 2006 is taken as training dataset (252 observations) and the data from January 2007 to February 2010 are chosen as testing dataset (38 observations). In order to evaluate the prediction capability of the proposed ensemble learning methodology, we carry out 1-step-ahead and 6-step-ahead prediction to test the short-term and long-term forecasting capability respectively, and 2-step-ahead and 3-step-ahead forecasts are performed for medium-term prediction capability evaluation.

For comparison of the forecasting performance of multiple different models, three criteria are used for evaluation of level prediction and directional forecasting. The mean absolute percent error (MAPE) and the root mean squared error (RMSE) are used as the evaluation criteria of the level prediction:

1

ˆ1 N t t

t

t

x xMAPE

N x=

−= � , ( )2

1

1ˆN

t ttRMSE x x

N == −�

(5) where ˆtx is the predicted value for tx , and N is the number of predictions.

In addition, the predict movement direction accuracy can be measured by a directional statistic (Dstat) [9], which can be expressed as

1

1100%

N

stat t

t

D aN =

= � (6)

where ta =1 if 1 1ˆ( )( ) 0t t t tx x x x+ +− − ≥ , and ta =0 otherwise.

To evaluate the forecasting ability of the proposed SD (M)-LSSVR-LSSVR ensemble learning model, this study compares its performance with some popular single forecasting models, such as Artificial Neural Networks ANN[14], Auto-Regressive Integrated Moving Average (ARIMA)[15], seasonal ARIMA (SARIMA)[15], SVR, LSSVR, and some variants of ensemble learning with other decomposition methods, e.g. empirical mode decomposition (EMD) [16] method and RMA(M) method , which are employed as benchmark models.

B. Experimental Results In the proposed SD (M)-LSSVR based decomposition-

and-ensemble method, the first step is to apply X-12-ARIMA algorithm to decompose the original hydropower consumption data series into seasonal factor, trend cycle and irregular component, as shown in Fig.3.

Jan85 Jan90 Jan95 Jan00 Jan05 Jan100

200

400

TC

Jan85 Jan90 Jan95 Jan00 Jan05 Jan100.5

1

1.5

SF

Jan85 Jan90 Jan95 Jan00 Jan05 Jan100.5

1

1.5

IR

Figure 3. The seasonal decomposition of hydropower consumption

The second step of the SD (M)-LSSVR based decomposition-and-ensemble method is the single forecasting of SF, TC and IR via LSSVR. Finally, the LSSVR model is used to combine the prediction results of each decomposed component into an ensemble forecasting result. In this study, we employ Gaussian RBF as the kernel function of LSSVR, and the values ofγ and 2σ parameters are first determined via 5-fold cross validation grid search method, and then they are adjusted by the trial-and-error approach.

For comparison purpose, some single forecasting models are also used. In these single prediction models, the ANN (I-H-O) in this study used I input neurons, ten hidden nodes and one output neuron. Here I is referred to lag time of the predicted data series [17]. The hidden nodes use logistic transfer function and the output layer uses the linear transfer function because the prediction performance is best when these transfer functions are used. In the single SVR and LSSVR models, the Gaussian RBF kernel function is also selected, and the values of parameters, i.e., γ , 2σ , ε are determined by cross validation method and the trial-and-

605

error method, the specified value of ε in SVR is set to 0.05. In the ARIMA (p-d-q) model and SARIM (p-d-q)(P-D-Q) model, the best model for each training sample is determined through Schwarz Criterion (SC) minimization.

Figs 4-6 show the performance comparisons between different models in all prediction horizons, respectively in terms of the MAPE, RMSE and Dstat criteria.

Figure 4. Performance comparisons for different methods in terms of

RMSE criteria

Figure 5. Performance comparisons for different methods in terms of MAPE criteria

Figure 6. Performance comparisons for different methods in terms of Dstat criteria

From Figures 4-6, it is easy to find that the proposed SD (M)-LSSVR-LSSVR method is the best one for hydropower consumption forecasting, relative to other models listed in this study, whether short-, medium- and long-term prediction. In all models listed in this study, the proposed SD (M)-LSSVR-LSSVR ensemble learning paradigm do not only achieve the highest accuracy from the level measurement, which is measured by the RMSE and MAPE criteria, but also gets the highest hit rate in direction decision, which is measured by Dstat criterion.

Among all of the models used in this study, the single ARIMA model performs the poorest no matter which step ahead. Not only has ARIMA model the lowest level accuracy which is measure by RMSE and MAPE, but also it gets the worst hit rate in direction decision, which is measured by Dstat criteria. The main reason is that ARIMA is a class of typical linear model and it cannot capture the nonlinear patterns and seasonal characteristic shown in the data series.

Apart from the proposed SD (M)-LSSVR-LSSVR method and ARIMA model which perform the best and the poorest in all the models listed in this study, other models produce some interestingly mix results. These results are analyzed from three accuracies (i.e. RMSE, MAPE and Dstat).

Firstly, in the case of level prediction accuracy, the results of RMSE criteria show that the EMD-LSSVR-LSSVR method rank after the proposed SD (M)-LSSVR-LSSVR model, and followed by RMA (M)-LSSVR model, the single AI model, SARIMA model and the poorest model is ARIMA model. Except for that the one-step-ahead prediction of EMD-LSSVR-LSSVR and three-step-ahead forecasts of RMA (M) are inferior to single LSSVR and ANN, the ensemble methods outperform single methods. The main reason could be that the decomposition strategy does effectively improve the prediction performance. Focusing on the single methods, all AI models and SARIMA model outperform ARIMA model. The reason may be that ARIMA is a typical linear model, which is not suitable for capturing the nonlinear and seasonal characteristic of hydropower consumption series. In AI models (i.e., LSSVR, ANN, SVR), it can be seen that ANN performs better than LSSVR except for 3-step-ahead prediction and LSSVR outperforms SVR model in all steps prediction. The main reason leading to this may be from the parameter selection. The SARIMA is slightly inferior to all AI models. The possible reason is that the prediction results of SARIMA which is under the linear hypothesis are more volatile than those of the AI models.

Second, in the case of another level accuracy of MAPE criteria, RMA (M)-LSSVR is the second to SD (M)-LSSVR-LSSVR, followed by ANN, LSSVR, EMD -LSSVR-LSSVR, SARIMA, SVR and ARIMA model. Unlike the results of RMSE criterion, EMD-LSSVR-LSSVR is inferior to some single AI models (i.e., ANN, LSSVR) with the exception of six-step-ahead prediction which it performs the second. The main reason leading to the abovementioned results is that the EMD method does not exactly catch the seasonal pattern of hydropower consumption. In addition, SARIMA outperforms slightly SVR other than one-step-ahead forecasts. The possible reason is that the SARIMA model can capture the seasonal pattern while the SVR can model the nonlinear factors hidden in the datasets. For this reason, one of them cannot be superior to the other one in all cases.

606

Third, the high level accuracy does not necessarily mean that there is a high hit rate in forecasting direction of hydropower consumption. The correct forecasting direction is important for policy manager to make investment plan in hydropower. Therefore the Dstat comparison is necessary. In Figures 4-6, it can be found that in terms of Dstat criterion, some similar conclusions can be drawn. First of all, the proposed SD (M)-LSSVR-LSSVR perform significantly better than all other models in all cases, followed by the other two ensemble models, then two of the single AI models(i.e., LSSVR, ANN), SARIMA, SVR, and lastly the ARIMA model. Second, the three ensemble hybrid methods mostly outperform the single prediction models. Furthermore, among the ensemble methods, SD (M)-LSSVR-LSSVR performs the best, and EMD-LSSVR-LSSVR outperforms RMA (M)-LSSVR-LSSVR model. Finally, LSSVR outperforms other single methods, SARIMA has the similar performance as ANN except for the one-step-ahead prediction and it outperforms SVR. The possible reason leading to this phenomenon may be the parameters choice. We also find that ARIMA model has the lowest directional accuracy without exception.

In addition, comparing different prediction horizons, the shorter the prediction horizon, the better the models

perform. Generally speaking, from the analysis of the experiments

presented in this study, we can draw the following several conclusions.

(1) The experimental results show that the proposed SD (M)-LSSVR-LSSVR method is significantly superior to the EMD-LSSVR-LSSVR model, RMA(M)-LSSVR, the single AI model (i.e., LSSVR, SVR, ANN) and the single SARIMA model, as well as the single ARIMA model, in terms of both level accuracy and direction accuracy.

(2) The prediction performance of the SD (M)-LSSVR-LSSVR, EMD-LSSVR-LSSVR and RMA (M)-LSSVR are generally better than the single methods. This indicates that the “divide-and-conquer” principle or “decomposition and ensemble” strategy can effectively improve the prediction performance in the case of hydropower consumption.

(3) The models applied in this study perform generally better in short-term prediction than in long-term forecasting.

(4) Due to the seasonal, nonlinear and non-stationary appearance of hydropower consumption, nonlinear models with seasonal adjustment are more suitable for prediction than linear methodologies. Thus, this leads to the fourth conclusion.

(5) The proposed SD (M)-LSSVR based decomposition-and-ensemble method, which is of effective seasonal decomposition and nonlinear prediction, can be used as a promising solution for the time series forecasting with nonlinear and seasonality.

IV. CONCLUSIONS Due to the intrinsic complexity of hydropower

consumption, a decomposition-and-ensemble methodology

integrating SD (M) and LSSVR based on the principle of “decomposition and ensemble” is proposed for the hydropower consumption forecasting. According to empirical study, we find that the proposed ensemble learning methodology can improve the prediction performance significantly and can outperform other popular used methods listed in this study in terms of the measure of level accuracy and direction accuracy. This indicates that the SD (M)-LSSVR based decomposition-and-ensemble method can be used as a very promising methodology for complex forecasting problems such as hydropower consumption forecasting with seasonal variations and nonlinearity.

REFERENCES [1] X.L.Chang, X.H. Liu, W. Zhou, Hydropower in China at present and

its further development. Energy, in press, 2009, doi:10.1016/j.energy.2009.06.051.

[2] C.T. Cheng, J.Y. Lin, Y.G. Sun, K. Chau, Long-Term Prediction of Discharges in Manwan Hydropower Using Adaptive-Network-Based Fuzzy Inference Systems Models. Advances in Natural Computation, vol. 3612, 2005, pp.1152-1161.

[3] C.H.R. Lima, U. Lall,Climate informed monthly streamflow forecasts for the Brazilian hydropower network using a periodic ridge regression model. Journal of Hydrology, vol. 380, 2010, pp.438-449.

[4] S. Zhou,X. Zhang,J. Liu, The trend of small hydropower development in China. Renewable Energy,vol. 34(4), 2009, pp.1078-1083

[5] V.S.Ediger, S. Akar, ARIMA forecasting of primary energy demand by fuel in Turkey. Energy Policy vol. 35(3), 2007, pp.1701-1708.

[6] P.Coulibaly, C.K. Baldwin, Nonstationary hydrological time series forecasting using nonlinear dynamic methods. Journal of Hydrology , vol. 307(1-4), 2005, pp.164-174.

[7] J.Z. Wang, W.J. Zhu, W.Y. Zhang, D.H. Sun, A trend fixed on firstly and seasonal adjustment model combined with the �-SVR for short-term forecasting of electricity demand. Energy Policy, vol. 37(11), 2009, pp.4901-4909.

[8] S.Y.Wang, L.Yu, and K. K.Lai, Crude oil price forecasting with TEI@I methodology. Journal of Systems Sciences and Complexity, vol.18 (2), 2005, pp.145-166.

[9] L.Yu, S.Y.Wang, and K. K. Lai, Forecasting crude oil price with an EMD-based neural network ensemble learning paradigm. Energy Economics, vol. 30 (5), 2008, pp.2623-2635.

[10] U.S. Census Bureau, X-12 ARIMA Reference Manual Version 0.3, 2009. See also: http://www.census.gov/ts/x12a/v03/x12adocV03.pdf .

[11] F. R. Macaulay, The Smoothing of Time Series. National Bureau of Economic Research, New York,1931.

[12] V. Vapnik, The nature of statistical learning theory. Springer, Verlag, New York, 1995.

[13] J. A. K. Suykens and J.Vandewalle, Least squares support vector machine classifiers. Neural Processing Letters, vol. 9 (3), 1999, pp. 293–300.

[14] K. Hornik, M. Stinchocombe, H.White, Multilayer feedforward networks are universal approximators. Neural Networks, vol. 2(5), 1989, pp.359-366.

[15] G. E. P. Box, and G. M. Jenkins, Time series analysis: Forecasting and control, second ed. Holden Day Press, San Francisco, 1976.

[16] N. E. Huang, Z.Shen, and S. R Long, The empirical mode decomposition and the Hilbert spectrum for nonlinear and nonstationary time series analysis. Proceedings of the Royal Society , vol 454A, 1998, pp. 903-99.

[17] H. White, Connectionist nonparametric regression: multilayer feedforward networks can learn arbitrary mappings. Neural Networks, vol. 3, 1990, pp. 535-549.

607