Topic Moment tensor inversion and moment tensor interpretation
A Practical Randomized CP Tensor...
Transcript of A Practical Randomized CP Tensor...
![Page 1: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/1.jpg)
A Practical Randomized CP Tensor DecompositionCasey Battaglino1, Grey Ballard2, and Tamara G. Kolda3
[email protected] [email protected] [email protected]
Georgia Tech Computational Sci. and Engr.
Wake Forest University Sandia National Labs
SIAM CSE 2017 Atlanta, GA
![Page 2: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/2.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Tensors
2
I1
I2
I3
I4 I4
I5
etc…
order 3 order 4 order 5order 2order 1 order N
modes
![Page 3: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/3.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Problem
3
` . . .``Matrix
Decomposition (SVD)
![Page 4: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/4.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Problem
4
` . . .``Tensor
Decomposition (CP)
` . . .``Matrix
Decomposition (SVD)
![Page 5: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/5.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CP: Multi-Way Data Analysis
5
A
neur
ons
data matrix
trial #1 trial #50 trial #100
. . . . . .
phasetria
ls
neur
ons
reconstruction
B
+ +≈
≈ + +
principal components analysis (PCA)
Canonical Polyadic (CP) tensor decomposition
trial phase
=
=
reconstruction
neuronfactors
functionalcell types
within trialdynamics
phasefactors
learningdynamics
trial factors
lasttrial
firsttrial
trialend
trialstart
19.6-44.4-86.9-52.4-44.384.1
-23.418.2-45.4-19.846.945.0
46.6-1.216.0-1.0-2.3-2.8
(source: Alex Williams, Stanford/Sandia)
PCA
CP
(2D)
(3D)
![Page 6: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/6.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Standard Method
6
Alternating Least Squares (CP-ALS)
Solve a sequence of least squares problems, one for each mode, repeat
``````
mode 1 mode 2 mode 3
A(1) A(2) A(3)
![Page 7: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/7.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Our Method
7
Randomized Least Squares
or “sketching”
![Page 8: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/8.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Contributions• We show how randomized sketching techniques
can extend from matrices to general dense tensors
• We demonstrate a novel randomized least squares algorithm for the CP decomposition, with a novel stopping condition
• This enables us to scale to much larger data sets
• In addition to speed, there is evidence that this randomization increases robustness!
8
![Page 9: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/9.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Example: Linear Regression
9
min�
}A� ´ b}�2x ` �1Find line satisfying:
y3
“x31
……
………
…
�1
�2
`"
b “ A� ` "
m
n
n
x
y
1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
50
60
px3, y3q
![Page 10: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/10.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Method of Normal Equations Solves for :�
min�
}A� ´ b} pATAq´1ATA� “ pATAq´1ATb� “ A:b
y “ �2 ` �1x
Example: Linear Regression
10
y3
“x31
……
………
…
�1
�2
`"
b “ A� ` "
m
n
n
1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
50
60
x
y
![Page 11: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/11.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
In MATLAB:
min�
}A� ´ b} � – Azb
Example: Linear Regression
11
y3
“x31
……
………
…
�1
�2
`"
b “ A� ` "
m
n
ny “ �2 ` �1x
1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
50
60
x
y
![Page 12: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/12.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
What if we sample points?(Uniform, Random)
Sampling Linear Regression
12
row-sampling operator
“�1
�2
Sb “ SA�
s
n1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
50
60
![Page 13: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/13.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Sample A and b (MATLAB):>> srows = randi(m,s,1); %sample s rows w. replacement >> As = A(srows, :); >> bs = b(srows); >> x = A \ b; >> xs = As \ bs;
Sampling Linear Regression
13
y “ �2 ` �1x
row-sampling operator
Sb “ SA�
“�1
�2s
n1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
50
60
![Page 14: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/14.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
(Fast Solution, Approximate)
Sampling Linear Regression
14
y “ �2 ` �1x
y “ �2 ` �1x
“�1
�2
Sb “ SA�
s
n
y3
“x31
……
………
…�1
�2
m
1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
50
60
1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
50
60b “ A�
Chol: Opsn2 ` n3q flops
QR: Opsn2q flops
Chol: Opmn2 ` n3q flops
QR: Opmn2q flops
m " n
m " s ° n
![Page 15: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/15.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
“ �1
�2
5 5
1 4 1 4
may be rank-deficient —> underdetermined!
“ �1
�2
5 6 8 5 9
1 4 1 5 1 7 1 4 1 8
Danger: Undetermined Sampling
15
SA
SA
A
![Page 16: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/16.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Underdetermined Sampling
16
“
Ax
b0 0 0 0 0 c0 0 0 0
m
n
m
n
Consider a row containing the only nonzero in its column
Given full-rank A:
![Page 17: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/17.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Underdetermined Sampling
17
Any sampling must contain that row or be rank-deficient.
Uniform sampling w/ replacement: s = O(mlogm) samples needed
“xSA b
0 0 0 0
(so s > m; sampling is useless here!)
“
Ax
b0 0 0 0 0 c0 0 0 0
m
n
m
n
n
s s
(Avron, Maymounkov & Toledo 2010)
Consider a row containing the only nonzero in its column
Given full-rank A:
![Page 18: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/18.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing
18
Idea: Transform A,b before sampling
“
Ax
b
n0 0 0 0 0 c0 0 0 0
? ?
n
m
(Ailon & Chazelle, 2006 / Drineas, Mahoney, Muthukrishnan, Sarlós 2007 / Rokhlin & Tygert 2008)
![Page 19: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/19.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing
19
Trick: mix up the rows
00000100
1 -1 1
-1 -1 1
-1 -1
(good)
![Page 20: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/20.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing
20
Trick: mix up the rows
11111111
80000000
problem with vectors that are sparse in the frequency domain…F
(bad)
![Page 21: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/21.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing
21
Trick: mix up the rows
1-1111
-11
-1
26
-222
-2-22
random sign flipping helps spread out smooth vectors
11111111
D
F
(good)
![Page 22: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/22.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing
22
“Fast Johnson-Lindenstrauss Transform”
— FFT— WHT— DHT
(Ailon & Chazelle, 2006)
1-1
-11
1-1
-1-1
“
1-1
-11
1-1
-1-1
x
bF A FD D
Randomly sign-flip rows, then perform one of:
Trick: mix up the rows
F
Orthogonal transformations; Change of basis
![Page 23: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/23.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing
23
“Fast Johnson-Lindenstrauss Transform”
— FFT— WHT— DHT
(Ailon & Chazelle, 2006)
Randomly sign-flip rows, then perform one of:
1-1
-11
1-1
-1-1
“
1-1
-11
1-1
-1-1
x
bF A FD D
Trick: mix up the rows
F
Opmnq OpmqOpmn logmq Opm logmq
A b
![Page 24: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/24.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing
24
“Fast Johnson-Lindenstrauss Transform”
— FFT— WHT— DHT
(Ailon & Chazelle, 2006)
Randomly sign-flip rows, then perform one of:
1-1
-11
1-1
-1-1
“
1-1
-11
1-1
-1-1
x
bF A FD D
Trick: mix up the rows
F
Opmnq OpmqOpmn logmq Opm logmq
A b
(other options include DCT, random Givens rotations, random orthogonal matrix)
![Page 25: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/25.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing
25
xA bS S“
We should then be able to sample a small number of rows …
“Fast Johnson-Lindenstrauss Transform”
— FFT— WHT— DHT
(Ailon & Chazelle, 2006)
Randomly sign-flip rows, then perform one of:
Trick: mix up the rows
F(other options include DCT, random Givens rotations, random orthogonal matrix)
![Page 26: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/26.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Coherence
26
µpAq “ max
i}Qi,˚}22
Given a matrix Q whose columns are an orthonormal basis for A:
e.g.
Coherence is the maximum leverage score:
incoherent coherent
n
m§ µpAq § 1
rQ,Rs “ QRpAq
e.g. Identity matrixe.g. Well-mixed matrix
(measuring correlation with the standard basis)
![Page 27: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/27.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
#Samples
27
µpAq “ 1 Ñ s “ Opm logmq
µpAq “ n
mÑ s “ Opn log nq
incoherent coherent
n
m§ µpAq § 1
m
n
(tiny)
(larger than input)
(Avron, Maymounkov & Toledo 2010)
(to recover a full-rank preconditioner)
![Page 28: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/28.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
“Faster Approximate Least Squares”
28
minx
}Ax ´ b}2mix
solve exactly
use as approximatesolution for original problemsample
x “ SFDAzSFDb
minx
}FDAx ´ FDb}2
minx
}SFDAx ´ SFDb}2
(Drineas, Mahoney, Muthukrishnan, Sarlós 2007/11)
![Page 29: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/29.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Sketching
29
(Avron, Maymounkov & Toledo 2010)
— Precondition using Randomized Least Squares — Then apply a standard iterative method (LSQR) — ~4X overall speedup over LAPACK
Has found practical uses, e.g. “Blendenpik”:
minx
}Ax ´ b}2 minx
}SFDAx ´ SFDb}2
![Page 30: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/30.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Sketching
30
Can we apply this technique to tensor applications?
I1
I2
I3
![Page 31: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/31.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
0 2 4 6 8 10 120.91
0.92
0.93
0.94
0.95
0.96
0.97
0.98
0.99
Yes
31
CPRAND CPRAND-FFT(our algorithms):
(standard algorithms)
Time(s)
FitCP-ALS(R)
CP-ALS(H)
300x300x300 Tensor, R=5, Noise = 1%
![Page 32: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/32.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CP Decomposition
32
` . . .``
` . . .``SVD:
3D CP:
X « X “Rÿ
r“1
ar ˝ br ˝ cr
R … number of components
![Page 33: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/33.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Deriving CP-ALS
33
X « X “Rÿ
r“1
ar ˝ br ˝ cr
` . . .``
R
Store factors in matrix form:
I
J
K
R R
A
R
B CI J K
a1
b1
c1
a1b1 c1
![Page 34: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/34.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Tensor Fibers
34
I
J
K
I
J
K
I
J
K
mode-3 fibersmode-1 fibers mode-2 fibers
![Page 35: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/35.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Matricization
35
Xp1q Xp2q Xp3q
…I …J …K
JK IKIJ
I
J
K
I
J
K
I
J
K
mode-3 fibersmode-1 fibers mode-2 fibers
aka unfolding
![Page 36: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/36.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Kronecker Product
36
A b B “
»
—–a11B ¨ ¨ ¨ a1nB...
. . ....
am1B ¨ ¨ ¨ amnB
fi
�fl ,
A1,1 A1,2
A2,1 A2,2
B1,1 B1,2
B2,1 B2,2b
B1,1 B1,2
B2,1 B2,2
B1,1 B1,2
B2,1 B2,2
B1,1 B1,2
B2,1 B2,2
B1,1 B1,2
B2,1 B2,2
A1,1 A1,2
A2,1 A2,2
every pair of scalar entries multiplied, in a block structure
![Page 37: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/37.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CP Decomposition & Khatri-Rao Product
37
The CP Representation:
Has matricized form:
X « X “Rÿ
r“1
ar ˝ br ˝ cr
A d B “ ra1 b b1,a2 b b2, . . . ,aK b bKs
b dc1 C C d BBb1 c1 b b1
“columnwise kronecker”
Xp1q « ApC d BqT
![Page 38: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/38.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CP Decomposition
38
minA
}Xp1q ´ ApC d BqT}FAllows us to express the least squares problem:
coefficient matrix
Matricized form:
(order 3, extends to order N)``
R
a1
b1c1
unknown
R R
A
R
B CI J K
Xp1q « ApC d BqT
![Page 39: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/39.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Alternating Least Squares
39
minA
}Xp1q ´ ApC d BqT}Fcoefficient
matrix
minA
}Xp1q ´ ApC d BqT}Funknown
Solve for each factor A, B, C in alternating fashion:
A – Xp1q { pC d BqT
(Transpose least squares, so forward-slash)
![Page 40: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/40.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Alternating Least Squares
40
A – Xp2q { pC d BqT
B – Xp2q { pC d AqT
C – Xp3q { pB d AqTorder 3:
A – Xp2qpC d Bq { pCTC ˚ BTBqB – Xp2qpC d Aq { pCTC ˚ ATAqC – Xp3qpB d Aq { pBTB ˚ ATAq
repeat
R ˆ R
special structure
Xp1q
Xp1q
![Page 41: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/41.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CP-ALS
41
Algorithm 1 CP-ALS
1: procedure CP-ALS(X, R) ô X P RIˆJˆK
2: Initialize factor matrices B,C3: repeat
4: A – Xp1qrpC d BqTs: “ Xp1qpC d BqpCTC ˚ B
TBq:
5: B – Xp2qrpC d AqTs: “ Xp2qpC d AqpCTC ˚ A
TAq:
6: C – Xp3qrpB d AqTs: “ Xp3qpB d AqpBTB ˚ A
TAq:
7: until termination criteria met
8: return factor matrices A,B,C9: end procedure
init: random or nvec terminate when fit stops changing:1 ´ }X ´ X}
}X}
![Page 42: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/42.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CPRAND
42
A – Xp1q { pC d BqT
CP-RAND: Just sample Xp1q and pC d BqT
A – Xp1qST { SpC d BqT
{…
Xp1q
…R
pC d BqTI
JKJK
![Page 43: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/43.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CPRAND
43
CP-RAND: Just sample Xp1q and pC d BqT. . . and hope that pC d BqT is incoherent. . .
A – Xp1q { pC d BqT
{…
Xp1q
…R
pC d BqT
A – Xp1qST { SpC d BqT
I
JKJK
![Page 44: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/44.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CPRAND
44
Algorithm 1 CPRAND
1: procedure CPRAND(X, R, S) ô X P RIˆJˆK
2: Initialize factor matrices B,C3: repeat
4: Define uniform random sampling operators SA,SB ,SC
5: A – Xp1qSTApSApC d BqTq:
6: B – Xp2qSTBpSBpC d AqTq:
7: C – Xp3qSTCpSCpB d CqTq:
8: until termination criteria met
9: return A,B,C10: end procedure
(Order 3 Example, extends to arbitrary order)
![Page 45: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/45.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CPRAND
45
Algorithm 1 CPRAND
1: procedure CPRAND(X, R, S) ô X P RIˆJˆK
2: Initialize factor matrices B,C3: repeat
4: Define uniform random sampling operators SA,SB ,SC
5: A – Xp1qSTApSApC d BqTq:
6: B – Xp2qSTBpSBpC d AqTq:
7: C – Xp3qSTCpSCpB d CqTq:
8: until termination criteria met
9: return A,B,C10: end procedure
Trick 1: Instead of matricizing X and then sampling,
sample fibers from X and then matricize the result.
(Order 3 Example, extends to arbitrary order)
![Page 46: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/46.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CPRAND
46
C B
R R
IBIC
ij
i
jci bj*
…… S
S(C�B)
Trick 2: We don’t need to form full KR-Product
Algorithm 1 CPRAND
1: procedure CPRAND(X, R, S) ô X P RIˆJˆK
2: Initialize factor matrices B,C3: repeat
4: Define uniform random sampling operators SA,SB ,SC
5: A – Xp1qSTApSKRpSA,C,BqTq: “ Xp1qST
ApSApC d BqTq:
6: B – Xp2qSTBpSKRpSB ,C,AqTq: “ Xp2qST
BpSBpC d AqTq:
7: C – Xp3qSTCpSKRpSC ,B,AqTq: “ Xp3qST
CpSCpB d CqTq:
8: until termination criteria met
9: return A,B,C10: end procedure
SKR: Sample KRP without forming
(Order 3 Example, extends to arbitrary order)
![Page 47: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/47.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CPRAND
47
No mixing performed here, yet converges in many cases.
Algorithm 1 CPRAND
1: procedure CPRAND(X, R, S) ô X P RIˆJˆK
2: Initialize factor matrices B,C3: repeat
4: Define uniform random sampling operators SA,SB ,SC
5: A – Xp1qSTApSKRpSA,C,BqTq: “ Xp1qST
ApSApC d BqTq:
6: B – Xp2qSTBpSKRpSB ,C,AqTq: “ Xp2qST
BpSBpC d AqTq:
7: C – Xp3qSTCpSKRpSC ,B,AqTq: “ Xp3qST
CpSCpB d CqTq:
8: until termination criteria met
9: return A,B,C10: end procedure
µpC d Bq § µpCqµpBqOur result:
(Order 3 Example, extends to arbitrary order)
![Page 48: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/48.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
µpA b Bq “ µpAqµpBq
A b B “ QARA b QBRB “ pQA b QBqpRA b RBq
QApi, :q b QBpj, :q
Lemma 1:
Proof:
QAbB RAbB
Each row of has formQAbB
}a b b} “ }a}}b}Proof follows from
Coherence of Kronecker Product
48
AB b CD “ pA b CqpB b Dqidentity:
![Page 49: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/49.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Lemma 2:
Proof:
µpA d Bq § µpAqµpBq
pQA b QBqpRA d RBq “ pQA b QBqQRRR “ QAdBRAdB.
qTi . . . row i of QA b QB
Coherence of KR Product
49
QAdB RAdB
li “ ||qTi QR|| “ ||QT
Rqi|| § ||QTR||||qi||
identity:AB d CD “ pA b CqpB d Dq
![Page 50: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/50.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing Tensors
50
— Space Efficient (A single mixed tensor, avoid forming full KRP)
— Time Efficient (Minimize mixing costs, sample sizes)
Goals:
For better convergence & guarantees, we need to mix
![Page 51: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/51.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing Tensors
51
R…
C d BJK
JK
I
Naive approach: mix Xp1q,C d B directly, then sample.
Xp1q
(Expensive)
A – Xp1qST { SpC d BqTCPRAND without mixing was:
![Page 52: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/52.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing Factors
52
R…
C d B
R R
A
R
B CI J K JK
Can we avoid forming the full KRP?
![Page 53: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/53.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
R
s
Sample Khatri Rao product without fully forming
Transform individual factors
Mixing Factors
53
pFCDCC d FBDBBq
Trick 3:
R R
A
R
B CI J K
A – Xp1qST { SpC d BqTCPRAND without mixing was:
![Page 54: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/54.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
pFCDC b FBDBqpC d BqpFCDCC d FBDBBq
“
Mixing Factors
54
Fact:
So the mixed factors expand:
apply to Xp1q (the LHS)
Xp1qpFCDC b FBDBq
AB d CD “ pA b CqpB d Dq
![Page 55: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/55.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mode-n Multiplication
55
X ˆn U “ UXpnq
I
J
K
mode-3 fibers
K
ˆ3 “ … K
IJK
ˆU U
“ X ˆ1 FCDC ˆ2 FBDB ˆ3 IA
IAXp1qpFCDC b FBDBq
![Page 56: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/56.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing Factors
56
DAFAXp1qpFCDC b FBDBqmatricize in one mode
D´1A F´1
A
sample rows
DAFAXp1qpFCDC b FBDBqST
solve least squares
DAFAXp1qpFCDC b FBDBqST pSZp1qTq:
Xp1q “
inverse mix the final, small, solution
. . .
X “ X ˆ3 FCDC ˆ2 FBDB ˆ1 FADA
A –
A –
Trick 4: Mix original tensor
only once:
![Page 57: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/57.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing Factors
57
DAFAXp1qpFCDC b FBDBqmatricize in one mode
D´1A F´1
A
sample rows
DAFAXp1qpFCDC b FBDBqST
solve least squares
DAFAXp1qpFCDC b FBDBqST pSZp1qTq:
Xp1q “
inverse mix the final, small, solution
. . .
X “ X ˆ3 FCDC ˆ2 FBDB ˆ1 FADA
A –
A –
Trick 4: Mix original tensor
only once:
![Page 58: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/58.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing Factors
58
DAFAXp1qpFCDC b FBDBqmatricize in one mode
D´1A F´1
A
sample rows
DAFAXp1qpFCDC b FBDBqST
solve least squares
DAFAXp1qpFCDC b FBDBqST pSZp1qTq:
Xp1q “
inverse mix the final, small, solution
. . .
X “ X ˆ3 FCDC ˆ2 FBDB ˆ1 FADA
A –
A –
Trick 4: Mix original tensor
only once:
![Page 59: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/59.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing Factors
59
DAFAXp1qpFCDC b FBDBqmatricize in one mode
D´1A F´1
A
sample rows
DAFAXp1qpFCDC b FBDBqST
solve least squares
DAFAXp1qpFCDC b FBDBqST pSZp1qTq:
Xp1q “
inverse mix the final, small, solution
. . .
X “ X ˆ3 FCDC ˆ2 FBDB ˆ1 FADA
A –
A –
Trick 4: Mix original tensor
only once:
![Page 60: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/60.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing Factors
60
DAFAXp1qpFCDC b FBDBqmatricize in one mode
D´1A F´1
A
sample rows
DAFAXp1qpFCDC b FBDBqST
solve least squares
DAFAXp1qpFCDC b FBDBqST pSZp1qTq:
Xp1q “
inverse mix the final, small, solution
X “ X ˆ3 FCDC ˆ2 FBDB ˆ1 FADA
AAA –
A –
Trick 4: Mix original tensor
only once:
![Page 61: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/61.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Mixing Factors
61
Trick 4: Mix original tensor
only once:
Summary:
efficiently sample S fibers from mode n of
before reshaping
efficiently sample S rows of KRP
X
solve small system of size S * R
inverse mix small result at end
X “ X ˆ3 FCDC ˆ2 FBDB ˆ1 FADA
(Trick 1)
(Trick 2)
mix individual factors(Trick 3)
Apnq – D´1n F´1
n pXpnqSTqrpSZpnqq˚s:
![Page 62: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/62.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Algorithm 3: CPRAND-MIX
62
Algorithm 1 CPRAND
1: procedure CPRAND(X, R, S) ô X P RIˆJˆK
2: Mix:
ˆX – Xˆ3FCDC ˆ2 FBDB ˆ1 FADA
3: Initialize factor matrices B,C4: repeat
5: Define uniform random sampling operators SA,SB ,SC
6: A – D
´1A F´1
A
”ˆ
Xp1qSTApSKRpSA,FCDCC,FBDBBqTq:
ı
7: B – D
´1B F´1
B
”ˆ
Xp2qSTBpSKRpSB ,FCDCC,FADAAqTq:
ı
8: C – D
´1C F´1
C
”ˆ
Xp3qSTCpSKRpSC ,FBDBB,FCDCCqTq:
ı
9: until termination criteria met
10: return A,B,C11: end procedure
(Trick 5)
(Order 3 Example, extends to arbitrary order)
argminx2Rn
kAx� bk2 = argminx2Rn
����
<(A)=(A)
�x�
<(b)=(b)
�����2
![Page 63: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/63.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Algorithm 4: CPRAND-PREMIX
63
If we use a real-valued orthogonal transform such as the DCT, We can simply mix, call CPRAND, then unmix the output.
Algorithm 1 CPRAND-PREMIX1: function CPRAND-PREMIX(X, R, S) . X 2 RI1⇥···⇥IN
2: Define random sign-flip operators Dm and orthogonal matrices Fm, m 2 { 1, . . . , N }3: Mix:
ˆX X⇥1 F1D1 ⇥ · · ·⇥N FNDN
4: [�, {ˆA(n)}] = CPRAND(
ˆX, R, S)5: for n = 1, . . . , N do
6: Unmix: A
(n)= DnFT
nˆ
A
(n)
7: end for
8: return factor matrices {A(n) }9: end function
![Page 64: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/64.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Stopping Criterion
64
I
J
K CP-ALS: expensive inner product
Trick 5:
1 ´ }X ´ X}}X} “ 1 ´
b}X}2 ` }X}2 ´ 2 † X, X °
}X}
![Page 65: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/65.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
I
J
K CP-ALS:
CPRAND:
I
J
K Uniformly sample : X
expensive inner product
I 1 Ä I ” rIs b rJs b rKs
1 ´ }X ´ X}}X} “ 1 ´
b}X}2 ` }X}2 ´ 2 † X, X °
}X}
Choose some subset of tensor entries
Stopping Criterion
65
Trick 5:
![Page 66: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/66.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
I
J
K CP-ALS:
CPRAND:
I
J
K Uniformly sample : X
expensive inner product
approximate fit
I 1 Ä I ” rIs b rJs b rKs
1 ´ }X ´ X}}X} “ 1 ´
a|I| ¨ mean t e2i | i P I u
}X} « 1 ´a
|I| ¨ mean t e2i | i P I 1 u}X}
1 ´ }X ´ X}}X} “ 1 ´
b}X}2 ` }X}2 ´ 2 † X, X °
}X}
Stopping Criterion
66
Trick 5:
![Page 67: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/67.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
CPRAND:
I
J
K Uniformly sample : X
approximate fit
I 1 Ä I ” rIs b rJs b rKs
1 ´ }X ´ X}}X} “ 1 ´
a|I| ¨ mean t e2i | i P I u
}X} « 1 ´a
|I| ¨ mean t e2i | i P I 1 u}X}
How many samples are needed? Since we are approximating a mean,
we can apply the Chernoff-Hoeffding bound
Stopping Criterion
67
Trick 5:
![Page 68: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/68.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
I
J
K approximate fit
1 ´ }X ´ X}}X} “ 1 ´
a|I| ¨ mean t e2i | i P I u
}X} « 1 ´a
|I| ¨ mean t e2i | i P I 1 u}X}
Very conservative bound:
Stopping Criterion
68
Trick 5:
P � 372µ2
max
In practice:
ˆP = 2
14yields error < 10
�3on synthetic data
(assuming ei is i.i.d)
Yields 95% confidence that the sampled fit is within 5% of true fit
#samples
(µmax
= max(e2i ))
![Page 69: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/69.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
I
J
K approximate fit
1 ´ }X ´ X}}X} “ 1 ´
a|I| ¨ mean t e2i | i P I u
}X} « 1 ´a
|I| ¨ mean t e2i | i P I 1 u}X}
Stopping Criterion
69
Trick 5:
Terminate after fit ceases to decrease after a certain number of iterations
![Page 70: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/70.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Experimental Setup
70
Environment— MATLAB R2016a & Tensor Toolbox v2.6. — Intel Xeon E5-2650 Ivy Bridge 2.0 GHz machine with 32 GB of memory
Data— Synthetic random tensor— Factors of rank Rtrue
— Factors are generated with collinearity— Observation is combined with Gaussian noise— “Score” is a measure of how well the original factors are recovered,
from 0 to 1
![Page 71: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/71.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Example Run
71
True Rank = 5, Target Rank = 5, Noise = 1%, Collinearity 0.9
![Page 72: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/72.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Per-Iteration Times
72
R “ 5 S “ 10R logR
![Page 73: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/73.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Stopping Criterion Speedup
73
(stopping condition with
ˆP = 2
14)
![Page 74: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/74.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Preprocessing Times
74
nvec on each mode-n unfolding vs. fft on fibers of each mode
![Page 75: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/75.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Experiment 1: order 4
75
Size: 90x90x90x90 Rtrue: 5 R: {5,6} Noise: 1% 600 Runs Collinearity: {0.5,0.9}
● ● ●●●● ●●● ● ●●● ● ●●● ● ●●● ●● ● ●●● ●●●● ●
●●●●●●●
0 10 20 30 40 50 60Time (s)
●●● ●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●● ●●●●●●●●●
● ● ●● ●● ● ●●● ●● ●●●●● ● ●●● ●●
●● ●●● ●● ●●●● ●● ●●● ●● ● ●●● ●●●● ● ●● ●●● ●●● ●●●●● ●●
● ● ●●● ● ●●●● ●● ● ● ●● ●
0.96 0.97 0.98 0.99 1.00Score
● ●●●●● ●● ●● ●●● ● ●●● ● ●●● ●● ● ●●● ● ●●● ● ●
● ●● ● ● ●
0 50 100 150 200Iterations until Termination
●●● ● ● ●● ●● ●●●●
●● ● ●●●●● ● ●●● ●●● ●●● ●● ●●●●●●●● ● ● ●●● ● ● ●●● ● ● ●● ●
● ●● ●●● ●●● ● ●●●●● ● ●● ●
0.980 0.982 0.984 0.986 0.988 0.990Fit
CPRAND−MIX CPRAND CP−ALS(R) CP−ALS(H)
![Page 76: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/76.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Experiment 1: order 4
76
Size: 90x90x90x90 Rtrue: 5 R: {5,6} Noise: 10% 600 Runs Collinearity: {0.5,0.9}
● ●●● ●● ●● ●● ●●● ●● ● ●● ●●●●● ●●●● ● ●● ●● ●
●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●● ●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●
●●●●●●●●●●●●●●●●●●● ●●●●●●
0 2 4 6 8Time (s)
0.2 0.4 0.6 0.8 1.0Score
● ●● ●● ●●●● ●● ● ●● ●●●● ●● ●●● ●●●●● ●●●● ●●●● ●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●● ●● ●●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●● ●● ●●●●●●● ● ●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●● ●●●● ●●●●
●● ●● ● ●●●●● ●●●●● ●●●
0 50 100 150 200Iterations until Termination
●
●● ●● ● ● ● ● ●●●●● ● ● ●●● ●● ● ● ●●● ●● ●●●●●●● ●●● ●●●●●● ● ● ●●● ●●●●●● ● ●●● ●●●● ● ● ●●● ●●● ●●● ● ●● ●●●● ●● ●●● ● ●
● ●●●
0.865 0.875 0.885 0.895Fit
CPRAND−MIX CPRAND CP−ALS(R) CP−ALS(H)
![Page 77: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/77.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Conclusion• CPRAND can surprisingly outperform CP-ALS in
both time and factor recovery (given the same termination criteria).
• CPRAND has very low per-iteration time, but may take more iterations until termination.
• CPRAND-FFT takes fewer iterations and samples at the cost of pre-processing time.
77
(Experiment 1)
![Page 78: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/78.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Experiment 2: Hazardous Gas Classification
78
72 sensors 25,900 time steps
300 experiments per gas (3 gases: CO, Acetaldehyde, Ammonia)
13.4 GB, dense
We replicate an experiment from Vervliet & Lauthauwer, 2016:
25900 x 72 x 900 Tensor
time
experiment
sens
orGoal: Classify which gas is released in each experiment
Here we use CPRAND without mixing!
![Page 79: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/79.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Experiment 2: Hazardous Gas Classification
79
Method Median Time (s) Median Fit Median Classification ErrorCPRAND 53.6 0.715 0.61%
CP-ALS(H) 578.4 0.724 0.67%CP-ALS(R) 204.7 0.724 0.67%
Factors Recovered
by CPRAND:
S = 1000, P = 214
(over 10 trials)
![Page 80: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/80.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Experiment 3: COIL-100 Image Set
80
We replicate an experiment from Zhou, Cichocki, and Xie (2013):
COIL-100: 100 objects, images of each from 72 different poses. Each image is RGB with size 128x128 pixels.
128 x 128 x 3 x 7200 Tensor (2.8 GB) image size
color channel
image id
http://www1.cs.columbia.edu/CAVE/software/softlib/coil-100.php
![Page 81: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/81.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Experiment 3: COIL-100 Image Set
81
# Samples (S) Median Speedup Median Fit400 8.38 0.674450 7.98 0.676500 6.63 0.677550 7.29 0.678600 4.75 0.680650 4.73 0.680700 4.77 0.680750 4.52 0.681800 3.70 0.682850 4.90 0.678900 4.95 0.679950 4.22 0.6821000 2.84 0.684
5 trials, R = 20, P = 214
Here we use CPRAND-FFT (mixing):
![Page 82: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/82.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Future Work
• Randomization in the Tucker model
• Randomization in sparse CP
82
(for scalability)
![Page 83: A Practical Randomized CP Tensor Decompositionperso.ens-lyon.fr/bora.ucar/tensors-cse17/battaglino-talk.pdf · A Practical Randomized CP Tensor Decomposition Casey Battaglino1, Grey](https://reader030.fdocuments.us/reader030/viewer/2022041000/5ea04df7b410ef18a7447d75/html5/thumbnails/83.jpg)
A Practical Randomized CP Tensor Decomposition SIAM CSE 2017. Atlanta, GA 03/02/2017
Contact
• Casey Battaglino ([email protected])
83