Download - Hybridisation of ARIMA and Non-linear ... - coin.wne.uw.edu.plcoin.wne.uw.edu.pl/sakowski/qfrg/wp-content/uploads/2013/10/pa.pdf · SP500 Lahmiri 2011 Probabilistic NN 6/36. ... 3

IntroductionResearch Objectives

DataMethodology

ResultsConclusions

Hybridisation of ARIMA and Non-linear MachineLearning Models for Time-series Forecasting

Piotr Arendarski

University of Warsaw

December 5, 2012

1 / 36


DataMethodology

ResultsConclusions

Table of contents

1 Introduction

2 Research Objectives

3 Data

4 Methodology

5 Results

6 Conclusions

2 / 36


DataMethodology

ResultsConclusions

Linear vs. Non-Linaer Models

Conventional statistical methods, the autoregressiveintegrated moving average (ARIMA) is extensively utilized inconstructing a forecasting model

ARIMA cannot be utilized to produce an accurate model forforecasting nonlinear time series

Machine Learning algorithms have been successfully utilized todevelop a nonlinear model for forecasting time series

Determining whether a linear or nonlinear model should befitted to a real-world data set is difficult

3 / 36


DataMethodology

ResultsConclusions

Linear Approach - ARIMA Model

Box and Jenkins (1976) developed the autoregressive movingaverage to predict time series

The ARIMA model is used for prediction non-stationary timeseries when linearity between variables is supposed

However, in many practical situations supposing linearity isnot valid

For this reason, ARIMA models do not produce effectiveresults when used for explaining and capturing nonlinearrelations of many real world problems, what results inincreased forecast error

4 / 36


DataMethodology

ResultsConclusions

Non-linear Approach - Machine Learning Methods

To deal with this problem, various non-linear approaches have beensuggested in the literature.

Kernel-based machine learning

Support vector machines

Evolutionary algorithm

Genetic Programming

Others machine learning methods

Artificial neural network (ANN) and others algorithms

5 / 36


DataMethodology

ResultsConclusions

Support vector machines (SVM) vs. other methods

ANN is known to overfit data unless cross-validation is appliedwhile SVM does not overfit data and curse of dimensionalityis avoided

Unlike neural network methods, the SVM approach does notattempt to control model complexity by keeping the numberof features small

SVM training always finds a global minimum

Forecasted subject: Author: Year: SVM outperformance:Financial TS Tay 2001 Back-propagation NNStock price index Kim 2003 BPNNFutures contracts Cao 2003 BPNNNikkei255 directions Huang 2005 BPNNSP500 Lahmiri 2011 Probabilistic NN

6 / 36


DataMethodology

ResultsConclusions

Genetic Programming vs. other methods

1 SVM and GP are powerful methods2 The empirical comparison is hardly found in the literature

Forecasted subject: Author: Year: GP outperformance:Stock index Saski 1999 ANNInsurance industry Sanz 2003 SVMEgyptian stock market El-Telbany 2004 ANNSoftware reliability Zhang 2006 ANNGDP China, US, Japan Li 2007 ARIMAHomes prices Kaboudan 2007 ANNWave forecasting Gaur 2008 ANNStock market Rajabioun 2008 ANNIndustrial production Klucik 2009 ARIMATransport energy demand Forouzanfara 2012 ANN

7 / 36


DataMethodology

ResultsConclusions

Hybrid Models vs. single models

Proposed Hybrid: Author: Year: Hybrid outperformance:SARIMA - BPNN Tseng 2000 SARIMA, BPNNARIMA - ANN Zhang 2003 ARIMA, ANNARIMA - SVM Pai 2005 ARIMA, SVMSARIMA - SVM Chen 2005 SVM, SARIMAARIMA - ANN Diaz 2008 ANN, ARIMAARIMA - ANN Robles 2008 ARIMA, ANNARIMA - ANN Valenzuela 2008 ARIMA, ANNARIMA - ERNN Aladag 2009 FFNN, ARIMAARIMA - ANN Flores 2009 ARIMA, ANNARIMA - ANN Faruk 2009 ARIMA, ANNARIMA - ANN Khashei 2010 ARIMA, ANN

8 / 36


DataMethodology

ResultsConclusions

Hybrid Models vs. single models cont.

1 Lee (2010) used non-polynomial activation function forGenetic Programming

Proposed Hybrid: Author: Year: Hybrid outperformance:ARIMA - ANN Areekul 2010 ARIMA, ANNARIMA - GP Lee 2010 ARIMA, ANN, GPARIMA - RBFN Shafie-khah 2011 ARIMA, ARIMA-WalewetARIMA - RBFN Khashei 2011 ARIMA, ANNSVR - ARIMA Chen 2011 ARIMA, BPNNARIMA - BPNN.GA Wang 2012 ARIMA, BPNNARFIMA - FNN Aladag 2012 ARFIMA, FNNARIMA - SVM Nie 2012 ARIMA and SVMs

9 / 36


DataMethodology

ResultsConclusions

Research Objectives

The proposed model1 Hybrid ARIMA - Genetic Programming, ARIMA - SVM

The benchmark models1 Hybrid ARIMA

Contribution1 Genetic Programming for Polynomial Regression -

hybridisation with ARIMA2 Different time series modeling, universality of the proposed

hybrid forecasting model

10 / 36


DataMethodology

ResultsConclusions

Data

Financial time series

British Pound / US Dolar (GU)

S&P 500 index futures

Wheat futures

Time frame

Weekly data

01.01.1970 - 30.09.2012 giving a total 2176 weeklyobservations

In-sample data (1976 observations)

Out-of-sample data (200 observations)

11 / 36


DataMethodology

ResultsConclusions

GBPUSD data

12 / 36


DataMethodology

ResultsConclusions

S&P 500 data

13 / 36


DataMethodology

ResultsConclusions

Wheat data

14 / 36


DataMethodology

ResultsConclusions

A hybrid model

A hybrid model: yt = Lt + Nt (1)where:

yt represents the original positive time series at time t;

Lt represents the linear component

Nt is the nonlinear component of the model

The residuals can be obtained using the ARIMA model:rt = yt + Lt (2)where

rt is estimated using such nonlinear methods as GP or SVM

15 / 36


DataMethodology

ResultsConclusions

A hybrid model cont.

Lt is the forecasted value of Lt and is estimated using theARIMA model

The residual can be rewritten as follows:rt = f (rt−1, rt−2, ..., rt−n) + εt (3)where

f (rt−1, rt−2, ..., rt−n) represents the nonlinear function thatis constructed using GP or SVMεt is the random error term.

The hybrid model for forecasting time series is: yt = Lt + Nt (4)

16 / 36


DataMethodology

ResultsConclusions

A hybrid model step-by-step

1 Step1. The ARIMA model is used to model the linearcomponent of time series. That is, Lt is obtained by using theARIMA model.

2 Step2. From Step 1, the residuals from the ARIMA model areobtained. The residuals are modeled by the GP and SVMmodels in equation(3). That is, Nt is the forecast value ofequation (3) by using GP and SVM

3 Step3. Using equation(4), the forecasts of the hybrid modelare obtained by adding the forecasted values of linear(ARIMA) and nonlinear (GP and SVM components).

17 / 36


DataMethodology

ResultsConclusions

ARIMA introduction

The basic idea of this forecasting approach is to predict futurevalues of time series yt using:

past values of the time series, yt−k

the past residuals from the model, εt−k

ARIMA models form an important part of the Box-Jenkinsapproach to time-series modelling.

When one of the three terms is zero, it’s usual to drop ”AR”,”I” or ”MA”. For example, ARIMA(0,1,0) is I(1), andARIMA(0,0,1) is MA(1).

18 / 36


DataMethodology

ResultsConclusions

ARIMA setup

Model identification

The appropriate ARIMA(p, d, q) model is obtained byapplying the Akaike Information Criterion (AIC) rule

Parameter estimation

Modeling diagnosis

Tests for white noise residuals indicate whether the residualseries contains additional information that might be utilizedby a more complex model

19 / 36


DataMethodology

ResultsConclusions

Representations of ARIMA models

GBPUSD

ARIMA(1,1,3)

S&P 500

ARIMA(3,1,2)

Wheat

ARIMA(1,1,2)

20 / 36


DataMethodology

ResultsConclusions

SVM introduction

Basic idea of Support Vector Machines

Optimal hyperplane for linearly separable patterns

Extend to patterns that are not linearly separable bytransformations of original data to map into new spaceKernel function

21 / 36


DataMethodology

ResultsConclusions

SVM setup

Sigma estimation

Automatic sigma estimation (sigest)

Default parameters

Kernel - Radial Basis kernel ”Gaussian”

Cost of constraints violation - 0.1

Epsilon in the insensitive-loss function - 0.1

22 / 36


DataMethodology

ResultsConclusions

Genetic programming intro

Genetic Programming (GP) tackles learning problems by means ofsearching a computer program space for the most probableprogram given a functionality specification. The search isperformed using an Evolutionary Algorithm.

Figure : The Evolutionary Algorithm cycle

23 / 36


DataMethodology

ResultsConclusions

Genetic Programming of polynomial models

The polynomial regression problem can be formulated asfollows: given training examples {(x1, yi ), . . . , (xN , yN)} ofexplanatory variables, that is vectorsxi = (xi1, xi2, . . . , xid) ∈ Rd , and corresponding responsevalues yi ∈ RGenetic Programming of polynomial models is a powerfulparadigm for learning well performing non-linear regressionmodels

References: Nikolaev,N., and Iba,H. (2002). Genetic Programming ofPolynomial Models for Financial Forecasting.

24 / 36


DataMethodology

ResultsConclusions

Genetic programming setup

As for the GP model, the input, output variables are:(yt−1, yt−2, yt−3, yt−4, yt−5, yt−6, yt−7, yt−8) and yt , respectively.

To reduce the forecast error of nonlinear component of ARIMA -GP model, the fitness function of GP is expressed as:

N∑i=1

(rt − rt)2

N

where rt represents the actual residual value, and the rt representsthe forecasted value of rt .

25 / 36


DataMethodology

ResultsConclusions

Representation ARIMA-GP Model

Hybrid ARIMA-GP model generated for S&P 500 time series:

yt = 0.1387 · yt−1 − 0.6742 · yt−2 − 0.1295 · yt−3 − 0.2285 · εt−1 + 0.7626 · εt−2 +sin(0.103602645916204 + rt−8) · (0.177070398354227 · rt−4 · rt−7)

26 / 36


DataMethodology

ResultsConclusions

Mean Square Error - GBPUSD data

Out-of-sample resultsARIMA ARIMA-SVM ARIMA-GP

1 month 0.00283 0.00272 0.002192 months 0.00260 0.00324 0.002556 months 0.00165 0.00179 0.0014912 months 0.00120 0.00134 0.0009024 months 0.00078 0.00085 0.00060All sample 0.00055 0.00061 0.00048

27 / 36


DataMethodology

ResultsConclusions

Mean Square Error - S&P 500 data



28 / 36


DataMethodology

ResultsConclusions

Mean Square Error - Wheat data



29 / 36


DataMethodology

ResultsConclusions

Average weekly returns - GBPUSD data


1 month 0.008 0.016 0.0252 months -0.004 0.012 0.0236 months 0.004 0.006 0.01912 months 0.010 0.001 0.00324 months -0.007 0.000 0.002All sample 0.000 0.000 0.001

30 / 36


DataMethodology

ResultsConclusions

Average weekly returns - S&P data



31 / 36


DataMethodology

ResultsConclusions

Average weekly returns - Wheat data


1 month 0.089 0.063 0.0712 months 0.057 0.038 0.0696 months 0.056 0.011 0.06012 months 0.042 0.009 0.05924 months 0.019 -0.002 0.010All sample -0.001 -0.001 0.004

32 / 36


DataMethodology

ResultsConclusions

Average weekly returns - All data



33 / 36


DataMethodology

ResultsConclusions

Average weekly return - All data

Average weekly return - methods comparison

34 / 36


DataMethodology

ResultsConclusions

Prediction Accuracy

Hybrid structure : ARIMA - GP outperforms ARIMA andARIMA - SVM across all the time series in terms of MSE

35 / 36


DataMethodology

ResultsConclusions

Trading Performance

Hybrid structure : ARIMA - GP outperforms ARIMA andARIMA - SVM across all the time series in terms of tradingperformance.

In general, weekly average returns decrease as time pass

There is a need to re-estimated the models

36 / 36