Research Matters Nick Higham February 25, 2009 Nick Higham ...higham/talks/an16_cvl.pdf · Black...

Post on 19-Oct-2020

2 views 0 download

Transcript of Research Matters Nick Higham February 25, 2009 Nick Higham ...higham/talks/an16_cvl.pdf · Black...

Research Matters

February 25, 2009

Nick HighamDirector of Research

School of Mathematics

1 / 6

Charlie Van Loanand the Matrix Exponential

Nick HighamSchool of Mathematics

The University of Manchester

http://www.maths.manchester.ac.uk/~higham@nhigham, nickhigham.wordpress.com

Numerical Linear and MultilinearAlgebra: Celebrating Charlie Van Loan

SIAM Annual Meeting, BostonJuly 13, 2016

Charlie and MATLAB

Pioneer of the use of MATLAB in teaching.Popularized the colon notation in numerical linear algebra.

Nick Higham Charlie Van Loan 3 / 22

Laguerre (1867):

Peano (1888):

eA in Applied Mathematics

Frazer, Duncan & Collar, Aerodynamics Division ofNPL: aircraft flutter, matrix structural analysis.

Elementary Matrices & Some Applications toDynamics and Differential Equations, 1938.Emphasizes importance of eA.

Arthur Roderick Collar, FRS(1908–1986): “First book to treatmatrices as a branch of appliedmathematics”.

Nick Higham Charlie Van Loan 5 / 22

University of Manchester Period, 1974–1975

Science Research Council Research Fellow

Charles F. Van Loan. A study of the matrixexponential. Numerical Analysis Report No. 10,University of Manchester, Manchester, UK,August 1975.

“I became interested in the matrix exponentialduring the preparation of a talk I gave on thesubject in 1974 here at Manchester. Since thenI have been motivated by the work of B.N.Parlett [20] and by C. B. Moler with his ‘n badways to compute the matrix exponential(n ≥ 9)’ ”.

Nick Higham Charlie Van Loan 6 / 22

Contents

Defines f (A) by Cauchy integral.

Gives explicit, divided difference-based formula forf (T ), T triangular.

Bounds and perturbation bounds for etA.

Backward error result for Padé approximants.

Extensive use of Schur decomposition.“one of the most basic tenets of numericalalgebra, namely, anything that the Jordandecomposition can do, the Schurdecomposition can do better!”

Nick Higham Charlie Van Loan 8 / 22

A Bibliography of the Matrix Exponential“Dear Exponential Freak,I collect facts, algorithms, and referencesabout eAt . If you have one that is missing fromthese references, please write to me.” (Undated.)

Nick Higham Charlie Van Loan 9 / 22

Sensitivity of the Exponential

SINUM, 1977.

Jordan and Schur-based bounds.Bounded the condition number. Showed conditionnumber is minimal for normal A.

Contemporary work by Kågström.

Open Question

When is the condition number of eA large?

Nick Higham Charlie Van Loan 10 / 22

Integrals

exp([

A11 A12

0 A22

])=

eA11

∫ 1

0eA11(1−s)A12eA22s ds

0 eA22

.CVL, IEEE Trans. Automat. Control (1978):use in reverse direction to evaluate integrals involving eA

via exponentials of bock triangular matrices.

4th most cited paper!

Nick Higham Charlie Van Loan 11 / 22

Nineteen Dubious Ways (1)

Moler and Van Loan, SIREV, 1978. A ∈ Cn×n:

Power series Limit Scaling and squaring

I + A +A2

2!+

A3

3!+ · · · lim

s→∞(I + A/s)s (eA/2s

)2s

Cauchy integral Jordan form Interpolation

12πi

∫Γ

ez(zI − A)−1 dz Zdiag(eJk )Z−1n∑

i=1

f [λ1, . . . , λi ]

i−1∏j=1

(A − λj I)

Differential system Schur form Padé approximation

Y ′(t) = AY (t), Y (0) = I Qdiag(eT )Q∗ pkm(A)qkm(A)−1

Nick Higham Charlie Van Loan 12 / 22

Nineteen Dubious Ways (2)

Scaling and squaring: eA =(eA/s

)s.Hump phenomenon: ‖eA‖ � ‖eA/s‖s, instability.Backward error analysis for Padé approximants.S&S: Padé more efficient than Taylor. How to choosePadé degree to achieve given accuracy at min cost.

Pros and cons of various other methods explained;emphasis on numerical stability.Republished in SIREV 2003 with “Method 20: Krylovspace methods” and new sections.Taken together, CVL’s most cited-paper: ≥ 2800citations.

Nick Higham Charlie Van Loan 13 / 22

Nineteen Dubious Ways (2)

Scaling and squaring: eA =(eA/s

)s.Hump phenomenon: ‖eA‖ � ‖eA/s‖s, instability.Backward error analysis for Padé approximants.S&S: Padé more efficient than Taylor. How to choosePadé degree to achieve given accuracy at min cost.Pros and cons of various other methods explained;emphasis on numerical stability.Republished in SIREV 2003 with “Method 20: Krylovspace methods” and new sections.Taken together, CVL’s most cited-paper: ≥ 2800citations.

Nick Higham Charlie Van Loan 13 / 22

History of the S&S Method (1)

J. D. Lawson (1967): exponential integrators for ODE.Suggests using

(e2−sA

)2s

= eA with diagonal Padéapproximants (uncond. stable).

Ward (1977): m = 8, ‖2−sA‖1 ≤ 1.

MATLAB expm up to 2005: m = 6, ‖2−sA‖∞ ≤ 1/2.

In 2004 I started the eA chapter of my book Functions ofMatrices: Theory and Computation . . .

Nick Higham Charlie Van Loan 14 / 22

History of the S&S Method (2)

H (2005): choose (s,m) at run-time based on sharper,“non-explicit” error bounds.

m 3 5 7 9 13θm 0.015 0.25 0.95 2.1 5.4

Algorithm

for m = [3 5 7 9 13]if ‖A‖1 ≤ θm, X = rm(A), quit, end

end

A← A/2s with s ≥ 0 minimal s.t. ‖A/2s‖1 ≤ θ13 = 5.4X = r13

2s by repeated squaring.

Faster, and in practice more accurate, than previous alg.

Nick Higham Charlie Van Loan 15 / 22

History of the S&S Method (2)

H (2005): choose (s,m) at run-time based on sharper,“non-explicit” error bounds.

m 3 5 7 9 13θm 0.015 0.25 0.95 2.1 5.4

Algorithm

for m = [3 5 7 9 13]if ‖A‖1 ≤ θm, X = rm(A), quit, end

end

A← A/2s with s ≥ 0 minimal s.t. ‖A/2s‖1 ≤ θ13 = 5.4X = r13

2s by repeated squaring.

Faster, and in practice more accurate, than previous alg.

Nick Higham Charlie Van Loan 15 / 22

History of the S&S Method (3)

Overscaling (Kenney & Laub, 1998; Dieci & Papini, 2000) :large ‖A‖ causes larger than necessary s to be chosen,with harmful effect on accuracy.Al-Mohy & H (2009) “avoid” via sharper bounds.

Lemma

If k = pm1 + qm2 with p,q ∈ N and m1,m2 ∈ N ∪ {0},

‖Ak‖1/k ≤ max(‖Ap‖1/p, ‖Aq‖1/q).

Take {p,q} = {r , r + 1} for k ≥ r(r − 1).

Improved expm in MATLAB 2015b.

Nick Higham Charlie Van Loan 16 / 22

Software Scene

Al-Mohy & H (2009) scaling & squaring algorithm for eA is in

Julia,MATLAB,NAG Library,R,SciPy.

H & Deadman, A Catalogue of Software for MatrixFunctions (2016).

Nick Higham Charlie Van Loan 17 / 22

Action of the Exponential

Compute eAb without first computing eA.

What was originally wanted by Lawson (1967), butproblems small there!Krylov methods commonly used.

Nick Higham Charlie Van Loan 18 / 22

The Sixth Dubious WayMoler & Van Loan (1978, 2003)

Nick Higham Charlie Van Loan 19 / 22

Al-Mohy & Higham (2011)—expmv

Exploit, for integer s,

eAB = (es−1A)sB = es−1Aes−1A · · · es−1A︸ ︷︷ ︸s times

B.

Choose s so Tm(s−1A) =∑m

j=0(s−1A)j

j!≈ es−1A. Then

Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B

yields Bs ≈ eAB.

Choose s and m using b’err & ‖Ak‖1/k approach.

Nick Higham Charlie Van Loan 20 / 22

Al-Mohy & Higham (2011)—expmv

Exploit, for integer s,

eAB = (es−1A)sB = es−1Aes−1A · · · es−1A︸ ︷︷ ︸s times

B.

Choose s so Tm(s−1A) =∑m

j=0(s−1A)j

j!≈ es−1A. Then

Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B

yields Bs ≈ eAB.

Choose s and m using b’err & ‖Ak‖1/k approach.

Nick Higham Charlie Van Loan 20 / 22

Advantages of expmv

Versus one-step ODE integrators:

Fully exploits the linearity of the ODE.Variable order, up to m = 55.Backward error based; ODE integrator controls local(forward) errors.Overscaling avoided.

Versus Krylov methods:

Very competitive in cost and storage.Cost dominated by matrix–vector multiplications.Black box: no need to choose/tune parameters.

Nick Higham Charlie Van Loan 21 / 22

Charlie in Three Bullets

Aficionado of the Matrix Exponential.

Master of Matrix Computations.

Baseball nut andfamily man."Reproduced with thepermission of theCommissioner of MajorLeague Baseball".c©1988

Nick Higham Charlie Van Loan 22 / 22

Charlie in Three Bullets

Aficionado of the Matrix Exponential.Master of Matrix Computations.

Baseball nut andfamily man."Reproduced with thepermission of theCommissioner of MajorLeague Baseball".c©1988

Nick Higham Charlie Van Loan 22 / 22

Charlie in Three Bullets

Aficionado of the Matrix Exponential.Master of Matrix Computations.

Baseball nut andfamily man."Reproduced with thepermission of theCommissioner of MajorLeague Baseball".c©1988

Nick Higham Charlie Van Loan 22 / 22

References I

A. H. Al-Mohy and N. J. Higham.Improved inverse scaling and squaring algorithms forthe matrix logarithm.SIAM J. Sci. Comput., 34(4):C153–C169, 2012.

M. Aprahamian and N. J. Higham.The matrix unwinding function, with an application tocomputing the matrix exponential.SIAM J. Matrix Anal. Appl., 35(1):88–109, 2014.

L. Dieci and A. Papini.Padé approximation for the exponential of a blocktriangular matrix.Linear Algebra Appl., 308:183–202, 2000.

Nick Higham Charlie Van Loan 1 / 5

References II

R. A. Frazer, W. J. Duncan, and A. R. Collar.Elementary Matrices and Some Applications toDynamics and Differential Equations.Cambridge University Press, Cambridge, UK, 1938.xviii+416 pp.1963 printing.

N. J. Higham.Functions of Matrices: Theory and Computation.Society for Industrial and Applied Mathematics,Philadelphia, PA, USA, 2008.ISBN 978-0-898716-46-7.xx+425 pp.

Nick Higham Charlie Van Loan 2 / 5

References III

N. J. Higham and A. H. Al-Mohy.Computing matrix functions.Acta Numerica, 19:159–208, 2010.

N. J. Higham and E. Deadman.A catalogue of software for matrix functions. Version2.0.MIMS EPrint 2016.3, Manchester Institute forMathematical Sciences, The University of Manchester,UK, Jan. 2016.22 pp.Updated March 2016.

Nick Higham Charlie Van Loan 3 / 5

References IV

N. J. Higham and F. Tisseur.A block algorithm for matrix 1-norm estimation, with anapplication to 1-norm pseudospectra.SIAM J. Matrix Anal. Appl., 21(4):1185–1201, 2000.

R. A. Horn and C. R. Johnson.Topics in Matrix Analysis.Cambridge University Press, Cambridge, UK, 1991.ISBN 0-521-30587-X.viii+607 pp.

C. S. Kenney and A. J. Laub.A Schur–Fréchet algorithm for computing the logarithmand exponential of a matrix.SIAM J. Matrix Anal. Appl., 19(3):640–663, 1998.

Nick Higham Charlie Van Loan 4 / 5

References V

C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix.SIAM Rev., 20(4):801–836, 1978.

C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix, twenty-five years later.SIAM Rev., 45(1):3–49, 2003.

Multiprecision Computing Toolbox.Advanpix, Tokyo.http://www.advanpix.com.

Nick Higham Charlie Van Loan 5 / 5