Time-Series Analysis on Multiperiodic Conditional Correlation by Sparse Covariance Selection
-
Upload
michael-lie -
Category
Science
-
view
388 -
download
2
Transcript of Time-Series Analysis on Multiperiodic Conditional Correlation by Sparse Covariance Selection
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Time-Series Analysis on MultiperiodicConditional Correlation by Sparse
Covariance Selection
Michael Lie1
1Prof. Suzuki Taiji Lab.,Faculty of Science,
Department of Information Science,Tokyo Institute of Technology, Japan
February 12, 2015
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Agenda
To propose of the new statistical model:Sparse Multiperiodic Covariance Selection (M-CovSel)To propose of optimization method through ADMM
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Covariance Selection
Covariance Selection
Sparse Covariance SelectionY1, · · · ,Yn ∼
i.i.d.Np(µ,Σ).
argminX�0
− ln det X + trace(SX ) + λ‖X‖1
Original idea: Dempster (1972)Application to Sparse and High-dimensional Matrices:Meinshausen and Bühlmann (2006)
Problem Formulation: Banerjee, Ghaoui and d’Aspremont(2008)Solution through graphical lasso model: Friedman, Hastieand Tibshirani (2008)Solution by ADMM method: Boyd (2011)
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Application
Application: Markowitz’s Portfolio Selection
Portfolio Selection (Markowitz, 1952)
minwσ2
p,w = w>Sw s.t. w>1 = 1 ∴ w =S−11
1>S−11.
Here, the inverse of empirical covariance S−1 is needed!
The existing Covariance Selection: fixed time⇒ Covariance Selection analysis over time series is needed!
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Intuition
Intuition
Figure: Existing Model
By estimating X, we can construct the portfolio.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Intuition
Figure: Our Model
Sij :=1n
∑k ,l
(yk ,i − µi)(yl,j − µj)>,
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Problem Formulation
Problem Formulation
Consider a stationary-time process such that the multiperiodicinverse covariance matrix X can be expressed as
X =
X11 X12 X13 · · · X1,TX>12 X22 X23 · · · X2,TX>13 X>23 X33 · · · X3,T
......
.... . .
...X>1,T X>2,T X>3,T · · · XT ,T
︸ ︷︷ ︸
Tp columns
Tprow
s
.
Assumption: X is stationary time-process, such thatXi,i+h = Xj,j+h for all i , j .
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Problem Formulation
Sparse Multiperiodic Covariance Selection (M-CovSel):
argminX�0
f (X) := argminX�0
{− ln det X +
∑i,j
trace(
S>ij Xij
)+
λ1∑i,j
∥∥Xij∥∥
1 + λ2∑i,j
∑k>i,l>j
∥∥Xij − Xkl∥∥2
2
}subject to Xi,i+h = Xj,j+h, ∀i , j .
`1 : ‖w‖1 =∑
i
|wi | `2 : ‖w‖2F =
∑i
|wi |2
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Problem Formulation
We separate our model into two parts:
f (X) ≡ g(X) + h(X)
g(X) = − ln det X +∑i,j
trace(
S>ij Xij
),
h(X) = λ1∑i,j
∥∥Xij∥∥
1 + λ2∑i,j
∑k>i,l>j
∥∥Xij − Xkl∥∥2
F .
g(X): twice differentiable and strictly convexh(X): convex but non-differentiable
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Problem Formulation
Auxiliary Variables
X =
X11 X12 X13 · · · X1,TX>12 X22 X23 · · · X2,TX>13 X>23 X33 · · · X3,T
......
.... . .
...X>1,T X>2,T X>3,T · · · XT ,T
bvec−→ X′ =
X11...
X1,TX22
...X2,T
...XT ,T
︸ ︷︷ ︸
p
numX×
p
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Problem Formulation
H: stationary time matrix
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Problem Formulation
All D: time-difference matrix
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Problem Formulation
Simplified D: time-difference matrix
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Problem Formulation
minimize g(X) + h(Z)
subject to
X′ = ZDX′ = ZHX′ = 0
⇐⇒ X = Z
whereg(X) = − ln det X +
∑i,j
trace(
S>ij Xij
),
h(Z) = λ1∑i,j
‖Z1‖1 + λ2∑i,j
‖Z2‖2F ,
X =
X′
DX′
HX′
, Z =
Z1Z20
.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Alternating Direction Method of Multiplier (ADMM)
Solving Through ADMM
Algorithm 1 Overview of ADMM1: for k = 0, 1, · · · do2: X-update:3: Compute W(0) = (X(0))−1.4: for t = 1, 2, · · · do5: Compute the direction using steepest gradient descent d = −∇G(X).6: Use an Armijo’s rule based step-size selection to get α such that
X(t+1) = X(t) + αd (t) is positive definite and the objective value suffi-ciently decreases.
7: Update X.8: end for9: Z-update:
10: Update Z1 : Z(k+1)1 = Sλ1/ρ((X
′)(k+1) + Y(k)
ρ)
11: Update Z2:Z(k+1)
2 =ρD(X′)(k+1) + Y(k)
2λ2 + ρ
12: Y-update: Y(k+1) = Y(k) + ρ(
X(k+1) − Z(k+1))
13: end for
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Alternating Direction Method of Multiplier (ADMM)
minimize g(X) + h(Z)
subject to
X = ZDX = ZHX = 0
⇐⇒ X = Z
Its augmented Lagrangian is
Lρ(X, Z,Y) = g(X) + h(Z) + (ρ/2)
∥∥∥∥X− Z +Yρ
∥∥∥∥2
F,
g(X) = − ln det X +∑i,j
trace(
S>ij Xij
),
h(Z) = λ1∑i,j
‖Z1‖1 + λ2∑i,j
‖Z2‖2F .
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Alternating Direction Method of Multiplier (ADMM)
1 X-update:
X(k+1) := argminX
(− ln det X +
∑i,j
trace(
S>ij Xij
)+ρ
2
∥∥∥∥∥X− Z(k) +Y(k)
ρ
∥∥∥∥∥2
F
),
2 Z-update:
Z(k+1) := argminZ
(λ1 ‖Z1‖1 + λ2 ‖Z2‖2F
+ρ
2
∥∥∥∥∥X(k+1) − Z +Y(k)
ρ
∥∥∥∥∥2
F
),
3 Y-update:
Y(k+1) := Y(k) + ρ(
X(k+1) − Z(k+1)).
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Alternating Direction Method of Multiplier (ADMM)
X Update
The solution of
X(k+1) := argminX
(− ln det X +
∑i,j
trace(
S>ij Xij
)+ρ
2
∥∥∥∥X− Z(k) +Y(k)
ρ
∥∥∥∥2
F
)is solved through steepest gradient descent and the algorithmis as given in Algorithm 1 of line 2-8.
Algorithm 2 X Update1: Compute W(0) = (X(0))−1.2: for t = 1, 2, · · · do3: Compute the direction using steepest gradient descent d = −∇G(X).4: Use an Armijo’s rule based step-size selection to get α such that
X(t+1) = X(t) + αd (t) is positive definite and the objective value suffi-ciently decreases.
5: Update X.6: end for
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Alternating Direction Method of Multiplier (ADMM)
Z Update
Z Update
Zk+1 := argminZ
(λ1‖Z1‖1 + λ2‖Z2‖2F
+ (ρ/2)
∥∥∥∥∥X(k+1) − Z +Y(k)
ρ
∥∥∥∥∥2
F
).
The equation above can be separated as two equations asbelow:
Z(k+1)1 := argmin
Z1
(λ1‖Z1‖1 + (ρ/2)‖(X′)(k+1) − Z1 + Yk
1/ρ‖2F)
Z(k+1)2 := argmin
Z2
(λ2‖Z2‖2F + (ρ/2)‖D(X′)(k+1) − Z2 + Yk
2/ρ‖2F)
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Alternating Direction Method of Multiplier (ADMM)
Solution of Z Update
Z(k+1)1 := argmin
Z1
(λ1‖Z1‖1 + (ρ/2)‖(X′)(k+1) − Z1 + Yk
1/ρ‖2F)
Z(k+1)2 := argmin
Z2
(λ2‖Z2‖2F + (ρ/2)‖D(X′)(k+1) − Z2 + Yk
2/ρ‖2F)
The solution of first solution is simply the soft-thresholdingfunction of
Z(k+1)1 = Sλ1/ρ
((X′)(k+1) +
Y(k)
ρ
)
and the solution of second solution is
Z(k+1)2 =
ρD(X′)(k+1) + Y(k)
2λ2 + ρ.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Numerical Results
Execution environment:Intel Core i7-4770 CPU @ 3.40GHz (8 CPUs)8GB RAMR ver. 3.3.65126.0OS Windows 7 Professional 64 bit (6.1. build 7601)
Verifying:Convergence SpeedSparsity of the estimates
using random data sets and real data.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
All D
Simplified D
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: Runtime of n = 10, λ1 = 0.01, λ2 = 0.01.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: (i) Objective Values, (ii) Primal Residuals, and (iii) DualResiduals of n = 10,T = 5, λ1 = 0.01, λ2 = 0.01.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: The sparsity pattern of estimates from the model ofn = 10,T = 5, λ1 = 0.01, λ2 = 0.01.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Analysis on real dataStock data of 50 randomly selected companies from NASDAQPeriod: 4 January 2011 to 31 December 2014
Tick Name SectorPDCO Patterson Companies, Inc. Health CareOMER Omeros Corporation Health CareHEAR Turtle Beach Corporation Consumer DurablesQBAK Qualstar Corporation TechnologyUTHR United Therapeutics Corporation Health CarePLCE The Children&39;s Place Retail Stores, Inc. Consumer ServicesSUSQ Susquehanna Bancshares, Inc. FinanceIDCC InterDigital, Inc. MiscellaneousELON Echelon Corporation TechnologyBGCP BGC Partners, Inc. FinanceMRGE Merge Healthcare Incorporated. TechnologyTISA Top Image Systems, Ltd. TechnologyIPXL Impax Laboratories, Inc. Health CareROVI Rovi Corporation MiscellaneousIBCP Independent Bank Corporation FinanceBABY Natus Medical Incorporated Health CareHFFC HF Financial Corp. FinanceISLE Isle of Capri Casinos, Inc. Consumer ServicesITIC Investors Title Company FinanceSLGN Silgan Holdings Inc. Consumer DurablesZIOP ZIOPHARM Oncology Inc Health CareMXIM Maxim Integrated Products, Inc. TechnologyNEPT Neptune Technologies & Bioresources Inc Health CareUTMD Utah Medical Products, Inc. Health Care
.
.
.
.
.
.
.
.
.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: (i) Objective Values, (ii) Primal Residuals, and (iii) DualResiduals of T = 5, λ1 = 0.01, λ2 = 0.01 from real stock data.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: The sparsity pattern of estimates from the model ofT = 5, λ1 = 0.01, λ2 = 0.01 from real stock data.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: The covariance matrix plot of estimates from the model ofT = 5, λ1 = 0.01, λ2 = 0.01 from real stock data.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: Negative covariance value of estimates from the model ofT = 5, λ1 = 0.01, λ2 = 0.01 from real stock data.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: Negative covariance value of estimates from the model ofT = 5, λ1 = 0.01, λ2 = 0.01 from real stock data (zoom on T = 1).
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: The weak positivity of estimates from the model ofT = 5, λ1 = 0.01, λ2 = 0.01 from real stock data.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Numerical Results
Figure: The weak positivity of estimates from the model ofT = 5, λ1 = 0.01, λ2 = 0.01 from real stock data (zoom on T = 1).
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
Conclusion and Discussion
Conclusions:ADMM algorithm with steepest gradient descent for Xupdate minimized our objective function f (X).Computation time took a lot of time as T increases.
Discussions:Instead of steepest gradient descent, Newton direction. cf.QUIC.Use Block Coordinate Descent as in BIG & QUIC.Introduce the decay constant in D.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
References I
[De72] Dempster, A. P. (1972). Covariance Selection. Biometrics 28 157-175.
[MB06] Meinshausen, N. and Bühlmann, P. (2006). High-dimensional graphs andvariable selection with the Lasso. Annals of Statistics 34 1436-1462.
[BG08] Banerjee, O., Ghaoui, E. L. and d’Aspremont, A. (2008). Model selectionthrough sparse maximum likelihood estimation for multivariate Gaussianor binary data. Journal of Machine Learning Research 9 485-516.
[Ti08] Friedman, J., Hastie, T. and Tibshirani, R. (2008). Sparse inversecovariance estimation with the graphical Lasso. Biostatistics 9 432-441.
[Ma52] Markowitz, H. (1952). Portfolio Selection. The Journal of Finance 7 77-91.
[Ti96] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso.Journal of the Royal Statistical Society: Series B 58 267-288.
[Bo11] Boyd, S., Parikh, N., Chu, E., Peleato, B. and Eckstein, J. (2011).Distributed optimization and statistical learning via the alternatingdirection method of multipliers. Foundations and Trends in MachineLearning 3 1-122.
Introduction Problem Setup Optimization Method Numerical Results Conclusion and Discussion References
References II
[Hs13] Hsieh, C. J., Sustik, M. A., Dhillon, I., Ravikumar, P. and Poldrack, R.(2013). BIG & QUIC: Sparse inverse covariance estimation for a millionvariables. In Advances in Neural Information Processing Systems3165-3173.
[Bv11] Bühlmann, P. and van de Geer, S. (2011). Statistics for High-DimensionalData: Methods, Theory and Applications. Springer-Verlag, Berlin.
[WB12] Wahlberg, B., Boyd, S., Annergren, M. and Wang, Y. (2012). An ADMMalgorithm for a class of total variation regularized estimation problems.ArXiv:1203.1828.