Explicit Link Between Periodic Covariance Functions and ...
Transcript of Explicit Link Between Periodic Covariance Functions and ...
![Page 1: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/1.jpg)
Explicit Link Between Periodic CovarianceFunctions and State Space Models
Arno Solin and Simo Sarkka
(AISTATS 2014)
Discussion by: Piyush Rai
December 12, 2014
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 2: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/2.jpg)
Overview
An explicit connection between GP regression with periodic covariancefunctions and state-space models
Based on expanding the periodic covariance function into a series ofstochastic resonators
Allows scaling up GP regression with periodic covariance functions to largedata sets
Proposed method also extended to GPs quasi-periodic covariance functions
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 3: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/3.jpg)
Gaussian Process Regression
The kernel view
Given: n training examples{(tk , yk)}, k = 1, . . . , n
yk = f (tk ) + ǫk
f (.) ∼ GP(0, k(t, t ′))
ǫk ∼ Nor(0, σ2n)
Prior assumptions about f (e.g.,smoothness, periodicity, etc.)encoded in the covariance functionk(t, t ′)
Can be solved in closed form butnaıve solution is expensive: O(n3)complexity at test time
The state-space view
Consider an m-th order SDE
df(t)
dt= Ff(t) + Lw(t)
yk = Hf(tk ) + ǫk
where f(t) contains derivatives off (t) up to order m − 1 and w(t) isthe white noise process withspectral density Qc
Model defined by F,L,Qc ,stationary covariance P∞, and theobservation model H
Solved using Kalman filtering andhas O(nm3) time-complexity
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 4: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/4.jpg)
Covariance and Spectral Density
- Shown here: a stationary covariance function (Matern) and its spectral density
- This equivalence enables transforming the GP into a state-space model and solvethe problem more efficiently in O(n) time
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 5: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/5.jpg)
GP Regression (the naıve way)
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 6: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/6.jpg)
GP Regression (the naıve way)
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 7: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/7.jpg)
GP Regression (via filtering and smoothing)
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 8: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/8.jpg)
GP Regression (via filtering and smoothing)
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 9: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/9.jpg)
This Paper
How to establish the GP vs state-space model equivalence when the GPcovariance function is periodic?
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 10: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/10.jpg)
Periodic Covariance Functions
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 11: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/11.jpg)
Periodic Covariance Functions
- Not amenable to the state-space transformation which requires the spectraldensity to be approximated by rational functions
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 12: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/12.jpg)
State-Space Formulation
Fourier series representation
kp(τ) =
∞∑
j=1
q2j cos(jω0τ)
Each periodic term j can be constructed as solution of a second-order ODE
(
xj (t)yj (t)
)
=
(
0 −ω0j
ω0j 0
)(
xj(t)yj(t)
)
with (xj (0), yj(0))⊤ ∼ Nor(0, q2j I)
One way to determine the coefficients q2j is via projection to the cosine basis
q2j =ω0
π
∫ +π/ω0
−π/ω0
kp(τ) cos(jω0τ)dτ
but there are other way too
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 13: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/13.jpg)
State-Space Formulation
The state-space model df(t)dt
= Ff(t) + Lw(t) will have block-diagonalmatrices F, L, and P∞ with blocks being (for j = 1, . . . , J)
Fpj =
(
0 −ω0j
ω0j 0
)
, Lpj = I2, Pp
∞,j = q2j I2
and the measurement model matrix H in yk = Hf(tk ) + ǫk is a block-rowvector of Hp
j = (1 0). The diffusion part is zero (deterministic model).
The spectral (variance) coefficients
q2j =2Ij(l
−2)
exp(l−2), for j = 1, 2, . . .
and q20 = I0(l−2)/ exp(l−2) where Iα(z) is the modified Bessel function
Taking the first J term of the series gives an approximation and thisapproximation converges uniformly to the actual covariance as J → ∞
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 14: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/14.jpg)
Approximated Covariance Functions (J=0)
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 15: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/15.jpg)
Approximated Covariance Functions (J=1)
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 16: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/16.jpg)
Approximated Covariance Functions (J=2)
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 17: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/17.jpg)
Approximated Covariance Functions (J=3)
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 18: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/18.jpg)
Quasi-Periodic Covariance Functions
The shape of the periodic effect may change with time
Modeled using a product of a truly periodic covariance function kp(t, t′) and
another covariance function kq(t, t′) with long characteristic length-scale
k(t, t ′) = kp(t, t′)kq(t, t
′)
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 19: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/19.jpg)
Quasi-Periodic Covariance Functions
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 20: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/20.jpg)
Quasi-Periodic Covariances: State-Space Form
Have state-space representations for both quasi and periodic parts
Set up the state-space such as the feedback matrices of both parts commute(i.e., FpFq = FqFp)
Properties of Kronecker product help accomplish this
The joint model for the quasi-periodic product of two covariance functionscan be written in a block-form
Fj = Fq ⊗ I2 + Iq ⊗ Fpj ,
Lj = Lq ⊗ Lpj ,
Qc,j = Qqc ⊗ q2j I2,
P∞,j = Pq∞
⊗ Pp∞,j ,
Hj = Hq ⊗Hpj
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 21: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/21.jpg)
Experiments: Computational Complexity
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 22: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/22.jpg)
Experiments: CO2 Concentration
Observations are CO2 concentrations across time (n = 2227)
GP covariance function
k(t, t ′) = kSE (t, t′) + kp(t, t
′)kν=3/2(t, t′) + kν=3/2(t, t
′)
Converted to state-space, hyperparams optimized w.r.t. marginal likelihood
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 23: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/23.jpg)
Experiments: Daily Births between 1969-1988
Observations are number of dailybirths between 1969-1988(n = 7305)
The model: Matern (ν = 7/2) forthe slow trend, Matern (ν = 3/2)for faster variation, Quasi-periodic(yearly) with Matern (ν = 3/2)damping, Quasi-periodic (weekly)with Matern (ν = 3/2) damping
Converted to state-space,hyperparams optimized w.r.t.marginal likelihood
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014
![Page 24: Explicit Link Between Periodic Covariance Functions and ...](https://reader031.fdocuments.us/reader031/viewer/2022012512/618b844ff7991747f25f67a4/html5/thumbnails/24.jpg)
Conclusions
Established connections between periodic covariance functions andstate-space models
The connection allows using efficient sequential inference methods (fromstate-space modeling) to solve periodic GP regression problem
Approximation error due to truncation available in closed-form
Solin and Simo Sarkka (AISTATS 2014) Periodic Covariance Functions & State Space Models December 12, 2014