Introduction to ANN & Fuzzy Systems
Lecture 10Time Series Model
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 2
Outline
• Time series models• Linear Time Series Models
– Moving Average Model – Auto-regressive Model – ARMA model
• Nonlinear Time Series Estimation
• Applications
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 3
Time Series • What is a time series?
– A scalar or vector-valued function of time indices
• Examples:– Stock prices– Temperature readings– Measured signals of all
kinds
• What is the use of a time series?– Prediction of future time
series values based on past observations
Modeling of a time seriesValues of a time series at
successive time indices are often correlated. Otherwise, prediction is impossible.
Most time series can be modeled mathematically as a wide-sense stationary (WSS) random process. The statistical properties do not change with respect to time.
Some time series exhibits chaotic nature. A chaotic time series can be described by a deterministic model but behaves as if it is random, and highly un-predictable.
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 4
Time Series Models
• Most time series are sampled from continuous time physical quantities at regular sampling intervals. One may label each such interval with an integer index. E.g.
{y(t); t = 0, 1, 2, …}. • A time series may have a
starting time, say t = 0. If so, it will have an initial value.
• In other applications, a time series may have been run for a while, and its past value can be traced back to t = .
• Notations– y(t): time series value at
present time index t. – y(t-1): time series value one
unit sample interval before t.
– y(t+1): the next value in the future.
• Basic assumption– y(t) can be predicted with
certainly degree of accuracy by its past values {y(tk); k > 0} and/or the present and past values of other time series such as {x(tm); m 0}
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 5
Time Series Prediction
• Problem StatementGiven {y(i); i = t1, …} estimate y(t+to), to 0 such that
is minimized. when to = 1, it is called a 1-step prediction.Sometimes, additional time series {u(i); i = t, t1, …} may be available to aid the prediction of y(i)
• The estimate of y(t + t0)that minimizes C is the conditional expectation given past value and other relevant time series.
• This conditional expectation can be modeled by a linear function (linear time series model) or a nonlinear function.
2
2
)(ˆ)(
)(
oo
o
ttyttyE
tteEC
1
ˆ( )
( ) |{ ( )} ,{ ( )}
o
t to i i
y t t
E y t t y i u i
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 6
A Dynamic Time Series Model
• State {x(t)}– Past values of a time series
can be summarized by a finite-dimensional state vector.
• Input {u(t)}– Time series that is not
dependent on {y(t)}
• The mapping is a dynamic system as y(t) depends on both present time inputs as well as past values.
))(),(()( tutxfty
Time series model
memory
y(t)
x(t)
Inputu(t)
x(t+1)
x(t) = [x(t) x(t1) … x(t p)] consists past values of {y(t)}u(t) = [u(t) u(t1) … u(t q)]
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 7
Linear Time Series Models
• y(t) is a linear combination of x(t) and/or u(t).
• White noise random process model of input {u(t)}:– E(u(t)) = 0
– E{u(t)u(s)} = 0 if t s; = s2 if t = s.
• Three popular linear time series models: 1. Moving Average (MA) Model:
2. Auto-Regressive (AR) Model:
3. Moving Average, Auto-regressive (ARMA) Model:
M
m
mtumbty0
)()()(
)()()()(1
tuntynatyN
n
M
m
N
n
mtumbntynaty01
)()()()()(
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 8
Moving Average Model
• Cross correlation function
• Auto-correlation function:
• An MA model is recognized by the finite number of non-zero auto-correlation lags.
• {b(m)} can be solved from {Ry(k)} using optimization procedure.
• Example: If {u(t)} is the stock price, then
is a moving average model – An average that moves with respect to time!
M
m
mtumbty0
)()()(
| |
2
0
( ) ( ) ( )
( ) ( ) | | ;
0 | |
y
M k
m
R k E y t y t k
b m b m k k M
k M
0
2
( ) ( ) ( )
( ) ( ) ( )
( ) 0 ,
0
if
otherwise.
yu
M
m
R i E y t u t i
b m E u t m u t i
b i i M
4
)3()2()1()()(
tututututy
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 9
Finding MA Coefficients• Problem: Given a MA time series
{y(t)}, how to find {b(m)} withtout knowing {u(t)}, except the knowledge of 2?
• One way to find the MA model coefficients {b(m)} is spectral factorization.
• Consider an example:
• Given {y(t); t = 1, …, T}, estimate auto-correlation lag
• For this MA(1) model, R(k)0 for k > 1.
• Power spectrum
• Spectral factorization:– Compute S(z) from {R(k)} and
factorize its zeros and poles to construct B(z)
• Or comparing the coefficients of polynomial and solve a set of nonlinear equations.
)1()1()()0()( tubtubty
kT
t
ktytykT
kR1
)()(1
)(ˆ
1
)1()0()1()0(
)()/1()()(
*
1*
zbb
z
bzb
zkRzBzBzSk
YY
B(z) B(1/z*)
zeros b(1)/b(0) (b(1)/b(0))*
poles 0
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 10
Auto-Regressive Model
• Ry(m) is replaced with R(m) to simplify notations.
• R is a Töeplitz matrix and is positive definite. Fast Cholesky factorization algorithms such as the Levinson algorithm can be devised to solve the Y-W equation effectively.
:formmatrix in or
001
:equationWalker -Yule the toleads This
)()()(
)()()()()(
)()()(
Hence,
)()(,0)(
)()()()(
22
1
2
1
2
1
eRa
T
N
ny
N
n
y
N
n
nkRnak
ktyntyEnaktytuE
ktytyEkR
tutyEtyE
tuntynaty
e2
=
aR
0001
1
01
10110 2σ
–a(N)
)–a(
)R()R(N–R(N)
)R(–N+)R()R(R(–N))R(–)R(
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 11
Auto-Regressive, Moving Average (ARMA) Model
• A combination of AR and MA model.
• DenoteThen
Thus,
M
m
N
n
mtumbntynaty01
)()()()()(
Ms
mtustyEmb
tvstyEM
m
for 0
)()()(
)()(
0
M
m
mtumbtv0
)()()(
.)()(
)()()()()(
)()()()()(
)()()(
1
01
01
MknkRna
mtuktyEmbnkRna
ktymtumbntynaE
ktytyEkR
N
ny
M
m
N
ny
M
m
N
n
y
if
)(
)1()(
)(
)1(
21
1121
LkR
kRkR
Na
a
N)LR(k)LR(k)LR(k
)R(k–N+)R(kR(k)R(k–N))R(k)R(k
=
Higher order Y-W equation
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 12
Nonlinear Time Series Model
• f(x(t), u(t)) is a nonlinear function or mapping:– MLP– RBF
• Time Lagged Neural Net (TLNN)– The input of a MLP network is
formed by a time-delayed segment of a time series.
• A neuronal filter
MLP
D D Du(tM)u(t1)
e(t)
)(ˆ ty
+
u(t)
y(t)
D
u(tM)
u(t1)
+
u(t)
D
D
f()
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 13
System Identification Problem
Consider an unknown system (Plant) with output y(t) which depends on current and past input u(t).
• System Identification Problem – Given: input u(t) and output y(t), 0 t tmax,
Find T[•] such that
u Unknown Plant
y
)()]([ˆ tytuT(t)y
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 14
Control Problem
Given: desired output y*(t), t1 t t2
Find: input u(t), t0 t t2 (t0 t1)
such that y(t) —> y*(t) for t1 t t2
• Path-Following Control Problem – Entire tragectory of the desired output sequence is specified (t1 ~ t0)
• Reinforcement Learning Problem – Only the destination is given. The intermediate path is not specified (t1 >> t0).
Introduction to ANN & Fuzzy Systems(C) 2001-2013 by Yu Hen Hu 15
System Identification
• With the same input {u(t)}, find a mathematical model which will best approximate the output sequence.
• Essentially, a function approximation problem. Due to the particular dynamics of the plant, recurrent ANN are often considered.
+Unknown Plant
Model
e(t)y(t)
y(t)
u(t)
^
Top Related