Prediction of a nonlinear time series with feedforward neural networks

Mats NikusProcess Control Laboratory

The time series

0 100 200 300 400 500 600 700 800 900 1000-1.5

-1

-0.5

0

0.5

1

1.5

2

k

y(k)

A closer look

250 260 270 280 290 300 310 320 330

-1

-0.5

0

0.5

1

1.5

k

y(k)

Another look

520 530 540 550 560 570

-1

-0.5

0

0.5

1

1.5

k

y(k)

Studying the time series

Some features seem to reapeat themselves over and over, but not totally ”deterministically”Lets study the autocovariance function

The autocovariance function

0 10 20 30 40 50 60 70 80 90 100-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

l (lag)

(l)

Studying the time series

The autocovariance function tells the same: There are certainly some dynamics in the dataLets now make a phaseplot of the dataIn a phaseplot the signal is plotted against itself with some lagWith one lag we get

Phase plot

-1.5 -1 -0.5 0 0.5 1 1.5 2-1.5

-1

-0.5

0

0.5

1

1.5

2

y(k)

y(k+

1)

3D phase plot

-2-1

01

2

-2

-1

0

1

2-2

-1

0

1

2

y(k-2)y(k-1)

y(k)

The phase plots tell

Use two lagged valuesThe first lagged value describes a parabolaLets make a neural network for prediction of the timeseries based on the findings.

The neural network

y(k+1)^

y(k) y(k-1)

Lets try with 3 hidden nodes2 for the ”parabola”and one for the ”rest”

Prediction results

0 5 10 15 20 25 30 35 40 45 50-1.5

-1

-0.5

0

0.5

1

1.5

2

sample #

y pred

ict y

Residuals (on test data)

0 50 100 150 200 250 300 350 400 450 500-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

sample #

y pred

ict-y

A more difficult case

If the time series is time variant (i.e. the dynamic behaviour changes over time) and the measurement data is noisy, the prediction task becomes more challenging.

Phase plot for a noisy timevariant case

-1.5 -1 -0.5 0 0.5 1 1.5 2-1.5

-1

-0.5

0

0.5

1

1.5

2

y(k-1)

y(k)

Residuals with the model

0 500 1000 1500 2000 2500-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

sample #

y pred

ict-y

Use a Kalman-filter to update the weights

We can improve the predictions by using a Kalman-filterAssume that the process we want to predict is described by

11

1

,

kkkk

kkk

vufy

w

Kalman-filter

Use the following recursive equations

kkk uNNy ,ˆˆ 1

1ˆˆ

k

kk

yC

111

RCPCCPK T

kkkTkkk

kkkkk yyK ˆˆˆ1

QPCKIP kkkk 1

The gradient needed inCk is fairly simple to calculate for a sigmoidalnetwork

Residuals

0 500 1000 1500 2000 2500-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

sample #

y pred

ict-y

Neural network parameters

0 500 1000 1500 2000 2500-30.5

-30

-29.5

-29

sample #

w10

0 500 1000 1500 2000 250035

40

45

50

sample #

w11

Henon series

The timeseries is actually described by

20

312

21

.b

.a

byyay kkk