Post on 19-Oct-2020
Research Matters
February 25, 2009
Nick HighamDirector of Research
School of Mathematics
1 / 6
Charlie Van Loanand the Matrix Exponential
Nick HighamSchool of Mathematics
The University of Manchester
http://www.maths.manchester.ac.uk/~higham@nhigham, nickhigham.wordpress.com
Numerical Linear and MultilinearAlgebra: Celebrating Charlie Van Loan
SIAM Annual Meeting, BostonJuly 13, 2016
Charlie and MATLAB
Pioneer of the use of MATLAB in teaching.Popularized the colon notation in numerical linear algebra.
Nick Higham Charlie Van Loan 3 / 22
Laguerre (1867):
Peano (1888):
eA in Applied Mathematics
Frazer, Duncan & Collar, Aerodynamics Division ofNPL: aircraft flutter, matrix structural analysis.
Elementary Matrices & Some Applications toDynamics and Differential Equations, 1938.Emphasizes importance of eA.
Arthur Roderick Collar, FRS(1908–1986): “First book to treatmatrices as a branch of appliedmathematics”.
Nick Higham Charlie Van Loan 5 / 22
University of Manchester Period, 1974–1975
Science Research Council Research Fellow
Charles F. Van Loan. A study of the matrixexponential. Numerical Analysis Report No. 10,University of Manchester, Manchester, UK,August 1975.
“I became interested in the matrix exponentialduring the preparation of a talk I gave on thesubject in 1974 here at Manchester. Since thenI have been motivated by the work of B.N.Parlett [20] and by C. B. Moler with his ‘n badways to compute the matrix exponential(n ≥ 9)’ ”.
Nick Higham Charlie Van Loan 6 / 22
Contents
Defines f (A) by Cauchy integral.
Gives explicit, divided difference-based formula forf (T ), T triangular.
Bounds and perturbation bounds for etA.
Backward error result for Padé approximants.
Extensive use of Schur decomposition.“one of the most basic tenets of numericalalgebra, namely, anything that the Jordandecomposition can do, the Schurdecomposition can do better!”
Nick Higham Charlie Van Loan 8 / 22
A Bibliography of the Matrix Exponential“Dear Exponential Freak,I collect facts, algorithms, and referencesabout eAt . If you have one that is missing fromthese references, please write to me.” (Undated.)
Nick Higham Charlie Van Loan 9 / 22
Sensitivity of the Exponential
SINUM, 1977.
Jordan and Schur-based bounds.Bounded the condition number. Showed conditionnumber is minimal for normal A.
Contemporary work by Kågström.
Open Question
When is the condition number of eA large?
Nick Higham Charlie Van Loan 10 / 22
Integrals
exp([
A11 A12
0 A22
])=
eA11
∫ 1
0eA11(1−s)A12eA22s ds
0 eA22
.CVL, IEEE Trans. Automat. Control (1978):use in reverse direction to evaluate integrals involving eA
via exponentials of bock triangular matrices.
4th most cited paper!
Nick Higham Charlie Van Loan 11 / 22
Nineteen Dubious Ways (1)
Moler and Van Loan, SIREV, 1978. A ∈ Cn×n:
Power series Limit Scaling and squaring
I + A +A2
2!+
A3
3!+ · · · lim
s→∞(I + A/s)s (eA/2s
)2s
Cauchy integral Jordan form Interpolation
12πi
∫Γ
ez(zI − A)−1 dz Zdiag(eJk )Z−1n∑
i=1
f [λ1, . . . , λi ]
i−1∏j=1
(A − λj I)
Differential system Schur form Padé approximation
Y ′(t) = AY (t), Y (0) = I Qdiag(eT )Q∗ pkm(A)qkm(A)−1
Nick Higham Charlie Van Loan 12 / 22
Nineteen Dubious Ways (2)
Scaling and squaring: eA =(eA/s
)s.Hump phenomenon: ‖eA‖ � ‖eA/s‖s, instability.Backward error analysis for Padé approximants.S&S: Padé more efficient than Taylor. How to choosePadé degree to achieve given accuracy at min cost.
Pros and cons of various other methods explained;emphasis on numerical stability.Republished in SIREV 2003 with “Method 20: Krylovspace methods” and new sections.Taken together, CVL’s most cited-paper: ≥ 2800citations.
Nick Higham Charlie Van Loan 13 / 22
Nineteen Dubious Ways (2)
Scaling and squaring: eA =(eA/s
)s.Hump phenomenon: ‖eA‖ � ‖eA/s‖s, instability.Backward error analysis for Padé approximants.S&S: Padé more efficient than Taylor. How to choosePadé degree to achieve given accuracy at min cost.Pros and cons of various other methods explained;emphasis on numerical stability.Republished in SIREV 2003 with “Method 20: Krylovspace methods” and new sections.Taken together, CVL’s most cited-paper: ≥ 2800citations.
Nick Higham Charlie Van Loan 13 / 22
History of the S&S Method (1)
J. D. Lawson (1967): exponential integrators for ODE.Suggests using
(e2−sA
)2s
= eA with diagonal Padéapproximants (uncond. stable).
Ward (1977): m = 8, ‖2−sA‖1 ≤ 1.
MATLAB expm up to 2005: m = 6, ‖2−sA‖∞ ≤ 1/2.
In 2004 I started the eA chapter of my book Functions ofMatrices: Theory and Computation . . .
Nick Higham Charlie Van Loan 14 / 22
History of the S&S Method (2)
H (2005): choose (s,m) at run-time based on sharper,“non-explicit” error bounds.
m 3 5 7 9 13θm 0.015 0.25 0.95 2.1 5.4
Algorithm
for m = [3 5 7 9 13]if ‖A‖1 ≤ θm, X = rm(A), quit, end
end
A← A/2s with s ≥ 0 minimal s.t. ‖A/2s‖1 ≤ θ13 = 5.4X = r13
2s by repeated squaring.
Faster, and in practice more accurate, than previous alg.
Nick Higham Charlie Van Loan 15 / 22
History of the S&S Method (2)
H (2005): choose (s,m) at run-time based on sharper,“non-explicit” error bounds.
m 3 5 7 9 13θm 0.015 0.25 0.95 2.1 5.4
Algorithm
for m = [3 5 7 9 13]if ‖A‖1 ≤ θm, X = rm(A), quit, end
end
A← A/2s with s ≥ 0 minimal s.t. ‖A/2s‖1 ≤ θ13 = 5.4X = r13
2s by repeated squaring.
Faster, and in practice more accurate, than previous alg.
Nick Higham Charlie Van Loan 15 / 22
History of the S&S Method (3)
Overscaling (Kenney & Laub, 1998; Dieci & Papini, 2000) :large ‖A‖ causes larger than necessary s to be chosen,with harmful effect on accuracy.Al-Mohy & H (2009) “avoid” via sharper bounds.
Lemma
If k = pm1 + qm2 with p,q ∈ N and m1,m2 ∈ N ∪ {0},
‖Ak‖1/k ≤ max(‖Ap‖1/p, ‖Aq‖1/q).
Take {p,q} = {r , r + 1} for k ≥ r(r − 1).
Improved expm in MATLAB 2015b.
Nick Higham Charlie Van Loan 16 / 22
Software Scene
Al-Mohy & H (2009) scaling & squaring algorithm for eA is in
Julia,MATLAB,NAG Library,R,SciPy.
H & Deadman, A Catalogue of Software for MatrixFunctions (2016).
Nick Higham Charlie Van Loan 17 / 22
Action of the Exponential
Compute eAb without first computing eA.
What was originally wanted by Lawson (1967), butproblems small there!Krylov methods commonly used.
Nick Higham Charlie Van Loan 18 / 22
The Sixth Dubious WayMoler & Van Loan (1978, 2003)
Nick Higham Charlie Van Loan 19 / 22
Al-Mohy & Higham (2011)—expmv
Exploit, for integer s,
eAB = (es−1A)sB = es−1Aes−1A · · · es−1A︸ ︷︷ ︸s times
B.
Choose s so Tm(s−1A) =∑m
j=0(s−1A)j
j!≈ es−1A. Then
Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B
yields Bs ≈ eAB.
Choose s and m using b’err & ‖Ak‖1/k approach.
Nick Higham Charlie Van Loan 20 / 22
Al-Mohy & Higham (2011)—expmv
Exploit, for integer s,
eAB = (es−1A)sB = es−1Aes−1A · · · es−1A︸ ︷︷ ︸s times
B.
Choose s so Tm(s−1A) =∑m
j=0(s−1A)j
j!≈ es−1A. Then
Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B
yields Bs ≈ eAB.
Choose s and m using b’err & ‖Ak‖1/k approach.
Nick Higham Charlie Van Loan 20 / 22
Advantages of expmv
Versus one-step ODE integrators:
Fully exploits the linearity of the ODE.Variable order, up to m = 55.Backward error based; ODE integrator controls local(forward) errors.Overscaling avoided.
Versus Krylov methods:
Very competitive in cost and storage.Cost dominated by matrix–vector multiplications.Black box: no need to choose/tune parameters.
Nick Higham Charlie Van Loan 21 / 22
Charlie in Three Bullets
Aficionado of the Matrix Exponential.
Master of Matrix Computations.
Baseball nut andfamily man."Reproduced with thepermission of theCommissioner of MajorLeague Baseball".c©1988
Nick Higham Charlie Van Loan 22 / 22
Charlie in Three Bullets
Aficionado of the Matrix Exponential.Master of Matrix Computations.
Baseball nut andfamily man."Reproduced with thepermission of theCommissioner of MajorLeague Baseball".c©1988
Nick Higham Charlie Van Loan 22 / 22
Charlie in Three Bullets
Aficionado of the Matrix Exponential.Master of Matrix Computations.
Baseball nut andfamily man."Reproduced with thepermission of theCommissioner of MajorLeague Baseball".c©1988
Nick Higham Charlie Van Loan 22 / 22
References I
A. H. Al-Mohy and N. J. Higham.Improved inverse scaling and squaring algorithms forthe matrix logarithm.SIAM J. Sci. Comput., 34(4):C153–C169, 2012.
M. Aprahamian and N. J. Higham.The matrix unwinding function, with an application tocomputing the matrix exponential.SIAM J. Matrix Anal. Appl., 35(1):88–109, 2014.
L. Dieci and A. Papini.Padé approximation for the exponential of a blocktriangular matrix.Linear Algebra Appl., 308:183–202, 2000.
Nick Higham Charlie Van Loan 1 / 5
References II
R. A. Frazer, W. J. Duncan, and A. R. Collar.Elementary Matrices and Some Applications toDynamics and Differential Equations.Cambridge University Press, Cambridge, UK, 1938.xviii+416 pp.1963 printing.
N. J. Higham.Functions of Matrices: Theory and Computation.Society for Industrial and Applied Mathematics,Philadelphia, PA, USA, 2008.ISBN 978-0-898716-46-7.xx+425 pp.
Nick Higham Charlie Van Loan 2 / 5
References III
N. J. Higham and A. H. Al-Mohy.Computing matrix functions.Acta Numerica, 19:159–208, 2010.
N. J. Higham and E. Deadman.A catalogue of software for matrix functions. Version2.0.MIMS EPrint 2016.3, Manchester Institute forMathematical Sciences, The University of Manchester,UK, Jan. 2016.22 pp.Updated March 2016.
Nick Higham Charlie Van Loan 3 / 5
References IV
N. J. Higham and F. Tisseur.A block algorithm for matrix 1-norm estimation, with anapplication to 1-norm pseudospectra.SIAM J. Matrix Anal. Appl., 21(4):1185–1201, 2000.
R. A. Horn and C. R. Johnson.Topics in Matrix Analysis.Cambridge University Press, Cambridge, UK, 1991.ISBN 0-521-30587-X.viii+607 pp.
C. S. Kenney and A. J. Laub.A Schur–Fréchet algorithm for computing the logarithmand exponential of a matrix.SIAM J. Matrix Anal. Appl., 19(3):640–663, 1998.
Nick Higham Charlie Van Loan 4 / 5
References V
C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix.SIAM Rev., 20(4):801–836, 1978.
C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix, twenty-five years later.SIAM Rev., 45(1):3–49, 2003.
Multiprecision Computing Toolbox.Advanpix, Tokyo.http://www.advanpix.com.
Nick Higham Charlie Van Loan 5 / 5