On computing the Jordan structure of Totally Nonnegative ... · On computing the Jordan structure...

On computing the Jordan structure of Totally NonnegativeMatrices with high relative accuracy

Teresa Laudadio (IAC–CNR Bari),Nicola Mastronardi (IAC–CNR Bari),

Paul Van Dooren ( UCLouvain)

Due giorni di Algebra Lineare NumericaRoma, 18–19 febbraio, 2019

On computing the Jordan structure of Totally NonnegativeMatrices with high relative accuracy

The revenge of LR method

Teresa Laudadio (IAC–CNR Bari),Nicola Mastronardi (IAC–CNR Bari),

Paul Van Dooren ( UCLouvain)

Due giorni di Algebra Lineare NumericaRoma, 18–19 febbraio, 2019

Table of contents

I Eigenvalue computation: LR vs QR method

I Totally Nonnegative (TN) matrices

I Computation of the Jordan form of TN matrices

I Numerical Examples

Eigenvalue computation: LR vs QR method

“... the QR algorithm is the ubiquitous tool for computingeigenvalues of dense matrices1. Its predecessor, the LR algorithm,is now largely forgotten and rarely taught to students. What wewish to say here is that, from an intellectual viewpoint, it was theLR algorithm that made the seminal contribution, and QR is animproved stable version of LR.”

1M. H. Gutknecht and B. N. Parlett, From qd to LR, or, how were the qdand LR algorithms discovered?, IMA Journal of Numerical Analysis, vol. 31,no. 3, pp. 741-754, 2011.

Eigenvalue computation: LR vs QR method

“... the QR algorithm is the ubiquitous tool for computingeigenvalues of dense matrices1. Its predecessor, the LR algorithm,is now largely forgotten and rarely taught to students. What wewish to say here is that, from an intellectual viewpoint, it was theLR algorithm that made the seminal contribution, and QR is animproved stable version of LR.”

1M. H. Gutknecht and B. N. Parlett, From qd to LR, or, how were the qdand LR algorithms discovered?, IMA Journal of Numerical Analysis, vol. 31,no. 3, pp. 741-754, 2011.

Eigenvalue computation: the LR method

I A1 = A ∈ Cn×n

for i=1,2,. . .LiRi = Ai

Ai+1 = RiLiend

I Li lower triangular, Li (j , j) = 1, j = 1, . . . , n,Ri upper triangular.

I If the factorization LiRi = Ai exists for any i , the matrices ofthe sequence {Ai}i∈N are similar to A.

I Theorem2. Let Fk = L1L2 · · · Lk . If the sequence {Fk}k∈Nconverges as k →∞, the the sequence {Ak}k∈N convergestoward an upper triangular matrix (symmetric case)

2H. R. Schwarz, H. Rutishauser, E. Stiefel, Numerical analysis of symmetricmatrices, Englewood Cliffs, N.J., Prentice-Hall, 1973

I A1 = A ∈ Cn×n

Ai+1 = RiLiend

I A1 = A ∈ Cn×n

Ai+1 = RiLiend

I A1 = A ∈ Cn×n

Ai+1 = RiLiend

Eigenvalue computation: the QR method

I A1 = A ∈ Rn×n

for i=1,2,. . .QiRi = Ai

Ai+1 = RiQi

I Qi unitary, Ri upper triangular.

I The matrices of the sequence {Ai}i∈N are similar to A.

I Teorema3. Sia A ∈ Cn×n con autovalori in modulo tuttidistinti: |λ1| > |λ2| > · · · > |λn| > 0. Sia X la matrice degliautovettori di A, tale che A = XΛX−1, e si supponga cheX−1 ammetta la fattorizzazione LU. Allora esistono matrici difase Sk tali che

limk→∞

SHk RkSk−1 = lim

k→∞SHk−1AkSk−1 = T , lim

k→∞SHk QkSk−1 = I .

3D. Bini, M. Capovani, O. Menchi, Metodi Numerici per l’Algebra Lineare,Zanichelli, 1988

I A1 = A ∈ Rn×n

Ai+1 = RiQi

limk→∞

SHk RkSk−1 = lim

I A1 = A ∈ Rn×n

Ai+1 = RiQi

limk→∞

SHk RkSk−1 = lim

I A1 = A ∈ Rn×n

Ai+1 = RiQi

limk→∞

SHk RkSk−1 = lim

LR vs QR method: stability and convergence

I The QR method is backward stable.

I The speed of convergence of both methods is linear.

I Shift strategies speed up the methods (quadraticconvergence) (cubic in the Hermitian case).

I Drawback: high relative accuracy can be lost in computingthe eigenvalues4.

I Recommendation : avoid subtraction between number withsame sign!

4J. Demmel,W. Kahan, Accurate Singular Values of Bidiagonal Matrices,1990, SIAM J. Sci. and Stat. Comput., 11(5), . 873-912.

Condition of an eigenvalue λ

The “condition” of the eigenvalue λ of A ∈ Cn×n is defined as

s(λ) =1

|yHx |

with x and y the right and left normalized eigenvectors of Aassociated to λ. “Roughly speaking, this means that O(ε)perturbation in A can induce ε/s(λ) changes in an eigenvalue5”

5G.H. Golub, C.F. Van Loan, Matrix Computations, 4th ed., The JohnsHopkins University Press Baltimore, 2013.

Condition of an eigenvalue λ

The “condition” of the eigenvalue λ of A ∈ Cn×n is defined as

s(λ) =1

|yHx |

with x and y the right and left normalized eigenvectors of Aassociated to λ. “Roughly speaking, this means that O(ε)perturbation in A can induce ε/s(λ) changes in an eigenvalue5”

5G.H. Golub, C.F. Van Loan, Matrix Computations, 4th ed., The JohnsHopkins University Press Baltimore, 2013.

Example: Bessel’s polynomials

Eigenvalues6 of a generalized Bessel matrix of order 20, B(4.5,2)20 :

(◦), exact by Mathematica,(·) approximations by Matlab with the matrix,

(+), approximations by Matlab with the matrix transpose.6L. Pasquini, Accurate computation of the zeros of the generalized Bessel polynomials, Numer. Math., 2000,

86, pp. 507-538.C. Ferreira, B. Parlett, F.M. Dopico, Sensitivity of eigenvalues of an unsymmetric tridiagonal matrix, Numer.Math., 2012, 122(3), pp. 527-555.

Condition of an eigenvalue λ. Hermitian caseTheorem7 Let A and A = A + E be Hermitian. Let theeigenvalues of A, A and E be

λ1 ≥ · · · ≥ λn, λ1 ≥ · · · ≥ λn, ε1 ≥ · · · ≥ εn.

Thenλ1 + εn ≤ λi ≤ λi + ε1, i = 1, . . . , n.

Consequently,

|λi − λi | ≤ ‖E‖2, i = 1, . . . , n. (1)

In the terminology of perturbation theory, (1) says that theeigenvalues of a Hermitain matrix are perfectly conditioned ...

unfortunately this does not imply that all the eigenvalues aredetermined to high relative accuracy. In fact, the relative error inλi is

|λi − λi ||λi |

≤ ‖E‖2|λi |

7G. W. Stewart, Matrix Algorithms: Vol. 2, Eigensystems, SIAM,Philadelphia, 2001

λ1 ≥ · · · ≥ λn, λ1 ≥ · · · ≥ λn, ε1 ≥ · · · ≥ εn.

Thenλ1 + εn ≤ λi ≤ λi + ε1, i = 1, . . . , n.

Consequently,

|λi − λi | ≤ ‖E‖2, i = 1, . . . , n. (1)

In the terminology of perturbation theory, (1) says that theeigenvalues of a Hermitain matrix are perfectly conditioned ...unfortunately this does not imply that all the eigenvalues aredetermined to high relative accuracy. In fact, the relative error inλi is

≤ ‖E‖2|λi |

λ1 ≥ · · · ≥ λn, λ1 ≥ · · · ≥ λn, ε1 ≥ · · · ≥ εn.

Thenλ1 + εn ≤ λi ≤ λi + ε1, i = 1, . . . , n.

Consequently,

|λi − λi | ≤ ‖E‖2, i = 1, . . . , n. (1)

In the terminology of perturbation theory, (1) says that theeigenvalues of a Hermitain matrix are perfectly conditioned ...unfortunately this does not imply that all the eigenvalues aredetermined to high relative accuracy. In fact, the relative error inλi is

≤ ‖E‖2|λi |

Totally Nonnegative Matrices: eigenvalue computation

Revenge of the LR method

A modification of LR method based on the Neville methodcomputes the eigenvalues of a totally nonnegative matrix A withhigh relative accuracy supposed A is factored as a product ofparticular bidiagonal matrices8

8P. Koev,Accurate eigenvalues and SVDs of totally nonnegative matrices,SIAM J. Matrix Anal. Appl., 27(1), pp. 1–23, 2005.

Totally Nonnegative Matrices: eigenvalue computation

Revenge of the LR method

A modification of LR method based on the Neville methodcomputes the eigenvalues of a totally nonnegative matrix A withhigh relative accuracy supposed A is factored as a product ofparticular bidiagonal matrices8

8P. Koev,Accurate eigenvalues and SVDs of totally nonnegative matrices,SIAM J. Matrix Anal. Appl., 27(1), pp. 1–23, 2005.

Totally Nonnegative Matrices

I A matrix is totally positive if each of its minors is positive.

I A matrix is totally nonnegative if each of its minors isnonnegative

I C. Cryer. Some properties of totally positive matrices. Linear Algebra Appl., 15,1–25, 1976.

I Shaun M. Fallat and Charles R. Johnson. Totally nonnegative matrices. Princeton Series in AppliedMathematics. Princeton University Press, Princeton, NJ, 2011.

I M. Gasca and J. M. Pena. Total positivity and Neville elimination. Linear Algebra Appl., 165,25–44, 1992.

I P. Koev, Accurate computations with totally nonnegative matrices. SIAM J. Matrix Anal. Appl.,29,731–751, 2007.

I P. Koev, Accurate Eigenvalues and Exact Zero Jordan Blocks of Totally Nonnegative Matrices, 2018.

Totally Nonnegative Matrices

I A matrix is totally positive if each of its minors is positive.

I A matrix is totally nonnegative if each of its minors isnonnegative

I C. Cryer. Some properties of totally positive matrices. Linear Algebra Appl., 15,1–25, 1976.

I Shaun M. Fallat and Charles R. Johnson. Totally nonnegative matrices. Princeton Series in AppliedMathematics. Princeton University Press, Princeton, NJ, 2011.

I M. Gasca and J. M. Pena. Total positivity and Neville elimination. Linear Algebra Appl., 165,25–44, 1992.

I P. Koev, Accurate computations with totally nonnegative matrices. SIAM J. Matrix Anal. Appl.,29,731–751, 2007.

I P. Koev, Accurate Eigenvalues and Exact Zero Jordan Blocks of Totally Nonnegative Matrices, 2018.

Totally nonnegative matrices

A totally nonnegative (TN) matrix can be factorized as theproduct of lower and upper bidiagonal (BD) TN matricesExample

1 2 41 3 91 4 16

1 21 3

= L2L1DU1U2

A totally nonnegative (TN) matrix can be factorized as theproduct of lower and upper bidiagonal (BD) TN matricesExample

1 2 41 3 91 4 16

1 21 3

= L2L1DU1U2

Storage of the entries of the bidiagonal matrices

Examples of Totally nonnegative matricesPascal matrix9 :

1 1 1 1 1 11 2 3 4 5 61 3 6 10 15 211 4 10 20 35 561 5 15 35 70 1261 6 21 56 126 252

Bidiagonal representation :

1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 1

9L. Aceto, D. Trigiante, The Matrices of Pascal and Other Greats, The

American Mathematical Monthly,108, 2001, pp. 232–245.

Examples of Totally nonnegative matricesPascal matrix9 :

1 1 1 1 1 11 2 3 4 5 61 3 6 10 15 211 4 10 20 35 561 5 15 35 70 1261 6 21 56 126 252

1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 1

9L. Aceto, D. Trigiante, The Matrices of Pascal and Other Greats, The

American Mathematical Monthly,108, 2001, pp. 232–245.

Example: Pascal matrices

46994838541.8026473129040614264498618434813808464 4.699483854180264e + 10 4.699483854180265e + 10554956959.8629315601218316654368527363981179735 5.549569598629313e + 08 5.549569598629270e + 0813982719.8768657579641932785550920189007397791 1.398271987686575e + 07 1.398271987686483e + 07

566741.0592514715502709682155302349922279150 5.667410592514719e + 05 5.667410592512758e + 0533061.3738535312656510304726125038782408082 3.306137385353126e + 04 3.306137385339706e + 042624.5205773818364234350310755556790308814 2.624520577381835e + 03 2.624520577264530e + 03275.3624643918710918790678389797005985188 2.753624643918711e + 02 2.753624643088091e + 0237.7838289328834004091152152207125557078 3.778382893288337e + 01 3.778382888204030e + 016.8607164608962072080859087566867594124 6.860716460896203e + 00 6.860716432387068e + 001.7487615421998684648886862656955835553 1.748761542199869e + 00 1.748761527349773e + 000.5718332522008930167280076975865210505 5.718332522008932e − 01 5.718332416239250e − 010.1457573717992378878694930414593258721 1.457573717992378e − 01 1.457573629317858e − 010.0264663489181133903334588757511631763 2.646634891811337e − 02 2.646634328298966e − 020.0036315770277857839944323164609232541 3.631577027785781e − 03 3.631574328494989e − 030.0003810219697334504586080098453478379 3.810219697334519e − 04 3.810212348803231e − 040.0000302467769316002152635002881173181 3.024677693160011e − 05 3.024699622580157e − 050.0000017644742403537149182711011678706 1.764474240353710e − 06 1.764921336162508e − 06

0.00000007151684427680540072441234011313 7.151684427680527e − 08 7.183315146869231e − 080.00000000180194154200172445069224455960 1.801941542001719e − 09 1.948297884304045e − 090.00000000002127893256001899627761173655 2.127893256001896e − 11 7.542297033530567e − 11

Exact eigenvalues (first column), Koev’s method (second column),QR method (eig of matlab) (third column).

Examples of Totally nonnegative matricesVariance–covariance (semi–separable) matrix of order-statistics fora random sample of size k from a population with exponentialdensity10

1k2 · · · 1

∑2j=1

1(k−j+1)2

∑2j=1

1(k−j+1)2

· · ·∑2

(k−j+1)2

∑2j=1

1(k−j+1)2

∑3j=1

1(k−j+1)2

· · ·∑3

(k−j+1)2

......

.... . .

...1k2

∑2j=1

1(k−j+1)2

∑3j=1

1(k−j+1)2

· · ·∑k

(k−j+1)2

1k2 1 · · · 1 11 1

(k−1)2...

F.A. Graybill, Matrices with Applications in Statistics, 2nd ed., Brooks/Cole, 1983.

Examples of Totally nonnegative matricesVariance–covariance (semi–separable) matrix of order-statistics fora random sample of size k from a population with exponentialdensity10

1k2 · · · 1

∑2j=1

1(k−j+1)2

∑2j=1

1(k−j+1)2

· · ·∑2

(k−j+1)2

∑2j=1

1(k−j+1)2

∑3j=1

1(k−j+1)2

· · ·∑3

(k−j+1)2

......

.... . .

...1k2

∑2j=1

1(k−j+1)2

∑3j=1

1(k−j+1)2

· · ·∑k

(k−j+1)2

1k2 1 · · · 1 11 1

(k−1)2...

F.A. Graybill, Matrices with Applications in Statistics, 2nd ed., Brooks/Cole, 1983.

Examples of Totally nonnegative matricesA correlation (Toeplitz semi–separable) matrix arising in the studyof Time Series and Markov chains is11

1 ρ ρ2 · · · ρn−2 ρn−1

ρ 1 ρ ρ2. . . ρn−2

ρ2 ρ 1. . .

. . ....

.... . .

. . .. . .

. . . ρ2

ρn−2. . .

. . . ρ 1 ρρn−1 ρn−2 · · · ρ2 ρ 1

1− ρ2 ρ ρ · · · ρρ 1− ρ2ρ 1− ρ2...

ρ 1− ρ2

11M.J.R. Healy, Matrices for Statistics, Oxford University press, 2000.

Examples of Totally nonnegative matricesA correlation (Toeplitz semi–separable) matrix arising in the studyof Time Series and Markov chains is11

1 ρ ρ2 · · · ρn−2 ρn−1

ρ 1 ρ ρ2. . . ρn−2

ρ2 ρ 1. . .

. . ....

.... . .

. . .. . .

. . . ρ2

ρn−2. . .

. . . ρ 1 ρρn−1 ρn−2 · · · ρ2 ρ 1

1− ρ2 ρ ρ · · · ρρ 1− ρ2ρ 1− ρ2...

ρ 1− ρ2

11M.J.R. Healy, Matrices for Statistics, Oxford University press, 2000.

Examples of Totally nonnegative matrices

Vandermonde matrices with ordered knots

1 1 · · · 1 1

x1 x2... xn−1 xn

x21 x22... x2n−1 x2n

......

xn−21 xn−22

... xn−2n−1 xn−2n

xn−11 xn−12

... xn−1n−1 xn−1n

x1 ≥ x2 ≥ · · · ≥ xn−1 ≥ xn.

Jordan blocks corresponding to zero eigenvalues

I n − rank(A) = # zero eigenvalues

I rank(Ai−1)− rank(Ai ) = # of Jordan blocks of size i

I A2 is TN (as a product of TN) and its BD is a TN-preservingop, thus BD accurate

I need to form BD of A2; . . . ,An, a potential O(n4) algorithm

Sketch of t Koev’s algorithm

r0 = n; i = 1;Ai = A;ri = rank(Ai );while i < n and ri−1 − ri > 0

i = i + 1;Ai = Ai−1 × A;ri = rank(Ai );

The proposed algorithm is based on the reduction of the TNmatrix A to a similar Hessenberg one via Neville eliminationsupposed A factored as a product of bidiagonal matrices.

Reduction of a TN matrix to Hessenberg form 1

Numerical exampleThe lower Hessenberg matrix

1 1 0 · · · 0

1 1 1. . .

.. . .

. . . 01 1 · · · 1 11 1 · · · 1 1

is irreducible and totally nonnegative with rank(Hn) = n − 1.Hn has

⌉distinct positive eigenvalues and exactly one Jordan

block of size⌊n2

⌋corresponding to the eigenvalue zero.

nz = 2090 5 10 15 20

Sparsity structure of the computed H20

Numerical example

3.918985947228994e + 00 3.918985947228999e + 00 + 0.000000000000000e + 00i3.682507065662362e + 00 3.682507065662364e + 00 + 0.000000000000000e + 00i3.309721467890570e + 00 3.309721467890570e + 00 + 0.000000000000000e + 00i2.830830026003772e + 00 2.830830026003772e + 00 + 0.000000000000000e + 00i2.284629676546570e + 00 2.284629676546571e + 00 + 0.000000000000000e + 00i1.715370323453430e + 00 1.715370323453430e + 00 + 0.000000000000000e + 00i1.169169973996227e + 00 1.169169973996176e + 00 + 0.000000000000000e + 00i6.902785321094297e − 01 6.902785321142250e − 01 + 0.000000000000000e + 00i3.174929343376376e − 01 3.174929292099200e − 01 + 0.000000000000000e + 00i8.101405277100522e − 02 8.199240254858542e − 02 + 0.000000000000000e + 00i0.000000000000000e + 00 5.116195905098703e − 02 + 2.135157259497554e − 02i0.000000000000000e + 00 5.116195905098703e − 02 − 2.135157259497554e − 02i0.000000000000000e + 00 2.537136779504252e − 02 + 4.389486455178316e − 02i0.000000000000000e + 00 2.537136779504252e − 02 − 4.389486455178316e − 02i0.000000000000000e + 00 −4.297088165307477e − 03 + 4.770120243833010e − 02i0.000000000000000e + 00 −4.297088165307477e − 03 − 4.770120243833010e − 02i0.000000000000000e + 00 −2.931310764823774e − 02 + 3.568142654713092e − 02i0.000000000000000e + 00 −2.931310764823774e − 02 − 3.568142654713092e − 02i0.000000000000000e + 00 −4.341230335978553e − 02 + 1.311956655234002e − 02i0.000000000000000e + 00 −4.341230335978553e − 02 − 1.311956655234002e − 02i

Computation of the eigenvalues of H20 : proposed method (firstcolumn), eig of matlab (second column).

Announcement 1

11th Workshop SDS2020STRUCTURAL DYNAMICAL SYSTEMS:

Computational AspectsPorto Giardino Resort

Capitolo (Monopoli) - Bari, ItalyJune 9-12, 2020

Local organizing committeeNicoletta del Buono et al.

Announcement 2

Householder Symposium 2020Hotel SIERRA SILVANA, Selva di Fasano (Br), Italy

June 14–19, 2020Local organizing committee

Nicola Mastronardi (chair) Dario A. BiniFasma Diele Teresa LaudadioCarmela Marangi Beatrice MeiniStefano Serra Capizzano Valeria Simoncini

Announcement 3

Sparse DaysSt Girons, France

either the week before or the week after the HouseholderSymposium

Local organizing committeeIain Duff et al.

On computing the Jordan structure of Totally Nonnegative ... · On computing the Jordan structure...

Documents

Transcript of On computing the Jordan structure of Totally Nonnegative ... · On computing the Jordan structure...

Separating Doubly Nonnegative and Completely Positive Matrices · n denote the cone of symmetric nonnegative n nmatrices. The cone of doubly nonnegative (DNN) matrices is then D n=

Convex and Semi-Nonnegative Matrix Factorizationsjordan/papers/ding-li-jordan-pami.pdfOct 24, 2008 · notably the singular value decomposition (SVD), which underlies principal component

Totally new, totally hot résumé

Nonnegative Ranks, Decompositions, and Factorizations of Nonnegative …lab.rockefeller.edu/cohenje/PDFs/211CohenRothblumLAA1993.pdf · 2019-09-18 · Nonnegative Ranks, Decompositions,

Topology of totally positive spaces - LACIMlacim.uqam.ca/~karp/lacim2020.pdfPostnikov (2006): totally nonnegative Grassmannian Gr 0 k;n. It has been related to the ASEP, the KP equation,

DSANLS: Accelerating Distributed Nonnegative Matrix ...nikos/wsdm2018.pdf · DSANLS: Accelerating Distributed Nonnegative Matrix ... (proximal coordinate descent), ... Accelerating

Weighted Nonnegative Matrix Factorization and Face Feature ...perso.uclouvain.be/paul.vandooren/publications/BlondelHV07.pdf · Weighted Nonnegative Matrix Factorization and Face

Nonnegative Matrix and Tensor Factorizations

Nonnegative Matrix Factorization - Complexity, Algorithms ... · Nonnegative Matrix Factorization, Neural Computation 2012. Householder XIX Nonnegative Matrix Factorization: Complexity,

Nonnegative matrix factorization with the Itakura-Saito ...kameoka/SAP/papers/Fevotte2009__Nonnegative_matrix...Nonnegative matrix factorization with the Itakura-Saito divergence.

COMPUTING A NONNEGATIVE MATRIX SANJEEV ARORA AND · COMPUTING A NONNEGATIVE MATRIX SANJEEV ARORA nonnegative matrix nonnegative matrices of size, denotes the Frobenius norm; we refer

PARAMETRIZATION OF THE MATRIX SYMPLECTIC GROUP AND ... · of totally nonnegative symplectic matrices, where we prove, in dimensions larger than two, that all its elements are diagonal

Improved tests and characterizations of totally ...garloff/Improved tests... · Improved Tests and Characterizations of Totally Nonnegative Matrices 589 given in [6]. However, in

Parametrizations of k-Nonnegative Matricespi.math.cornell.edu/~brosowsky/Media/UMN-final-presentation.pdf · In 1999, Fomin and Zelevinsky studied totally nonnegative matrices. They

Maximum Norms & Nonnegative Matrices

Thomas Lam - University of Michigantfylam/SLCLecture1.pdf · 2019-04-15 · Totally positive matrices De nition A real matrix A is totally positive (resp. totally nonnegative) if

Robust Nonnegative Matrix Factorization

Exploring Nonnegative Matrix Factorization - Stanford University

Initializations for Nonnegative Matrix Factorizationslrace/nmfinitializations.pdf · Initializations for Nonnegative Matrix Factorization Shaina Race North Carolina State University

Informed Nonnegative Matrix Factorization Methods for ...