Numerical Matrix AnalysisSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlNumerical Matrix AnalysisLinear Systems and Least SquaresIlse C. F. IpsenNorth Carolina State UniversityRaleigh, North CarolinaSociety for Industrial and Applied MathematicsPhiladelphiaSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlCopyright 2009 by the Society for Industrial and Applied Mathematics 10 9 8 7 6 5 4 3 2 1All rights reserved. Printed in the United States of America. No part of this book may bereproduced, stored, or transmitted in any manner without the written permission of thepublisher. For information, write to the Society for Industrial and Applied Mathematics, 3600 Market Street, 6th Floor, Philadelphia, PA, 19104-2688 USA.Trademarked names may be used in this book without the inclusion of a trademarksymbol. These names are used in an editorial context only; no infringement of trademarkis intended.Library of Congress Cataloging-in-Publication DataIpsen, Ilse C. F.Numerical matrix analysis : linear systems and least squares / Ilse C. F. Ipsen.p. cm.Includes index.ISBN 978-0-898716-76-41.Least squares. 2.Linear systems. 3.Matrices.I. Title. QA275.I67 2009511'.42--dc222009008514is a registered trademark.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page viiiiiiiiiiContentsPreface ixIntroduction xiii1 Matrices 11.1 What Is a Matrix? . . . . . . . . . . . . . . . . . . . . . . . 11.2 Scalar Matrix Multiplication . . . . . . . . . . . . . . . . . . 41.3 Matrix Addition . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Inner Product (Dot Product) . . . . . . . . . . . . . . . . . . 51.5 Matrix Vector Multiplication . . . . . . . . . . . . . . . . . 61.6 Outer Product . . . . . . . . . . . . . . . . . . . . . . . . . 81.7 Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . 91.8 Transpose and Conjugate Transpose . . . . . . . . . . . . . . 121.9 Inner and Outer Products, Again . . . . . . . . . . . . . . . 141.10 Symmetric and Hermitian Matrices . . . . . . . . . . . . . . 151.11 Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161.12 Unitary and Orthogonal Matrices . . . . . . . . . . . . . . . 191.13 Triangular Matrices . . . . . . . . . . . . . . . . . . . . . . 201.14 Diagonal Matrices . . . . . . . . . . . . . . . . . . . . . . . 222 Sensitivity, Errors, and Norms 232.1 Sensitivity and Conditioning . . . . . . . . . . . . . . . . . 232.2 Absolute and Relative Errors . . . . . . . . . . . . . . . . . 252.3 Floating Point Arithmetic . . . . . . . . . . . . . . . . . . . 262.4 Conditioning of Subtraction. . . . . . . . . . . . . . . . . . 272.5 Vector Norms . . . . . . . . . . . . . . . . . . . . . . . . . 292.6 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . 332.7 Conditioning of Matrix Addition and Multiplication . . . . . 372.8 Conditioning of Matrix Inversion . . . . . . . . . . . . . . . 383 Linear Systems 433.1 The Meaning of Ax =b . . . . . . . . . . . . . . . . . . . . 433.2 Conditioning of Linear Systems. . . . . . . . . . . . . . . . 443.3 Solution of Triangular Systems . . . . . . . . . . . . . . . . 513.4 Stability of Direct Methods . . . . . . . . . . . . . . . . . . 52viiSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page viiiiiiiiiiiviii Contents3.5 LU Factorization . . . . . . . . . . . . . . . . . . . . . . . . 583.6 Cholesky Factorization . . . . . . . . . . . . . . . . . . . . 633.7 QR Factorization . . . . . . . . . . . . . . . . . . . . . . . . 683.8 QR Factorization of Tall and Skinny Matrices . . . . . . . . 734 Singular Value Decomposition 774.1 Extreme Singular Values . . . . . . . . . . . . . . . . . . . . 794.2 Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 814.3 Singular Vectors . . . . . . . . . . . . . . . . . . . . . . . . 865 Least Squares Problems 915.1 Solutions of Least Squares Problems . . . . . . . . . . . . . 915.2 Conditioning of Least Squares Problems . . . . . . . . . . . 955.3 Computation of Full Rank Least Squares Problems . . . . . . 1016 Subspaces 1056.1 Spaces of Matrix Products. . . . . . . . . . . . . . . . . . . 1066.2 Dimension . . . . . . . . . . . . . . . . . . . . . . . . . . . 1096.3 Intersection and Sum of Subspaces . . . . . . . . . . . . . . 1116.4 Direct Sums and Orthogonal Subspaces . . . . . . . . . . . . 1156.5 Bases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118Index 121Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page ixiiiiiiiiPrefaceThis book was written for a rst-semester graduate course in matrix theory at NorthCarolina State University. The students come from applied and pure mathematics,all areas of engineering, and operations research. The book is self-contained. Themain topics covered in detail are linear system solution, least squares problems,and singular value decomposition.Myobjectivewastopresent matrixanalysisinthecontext ofnumericalcomputation, with numerical conditioning of problems, and numerical stability ofalgorithms at the forefront. I tried to present the material at a basic level, but in amathematically rigorous fashion.Main Features. This book differs in several regards from other numerical linearalgebra textbooks.Systematic development of numerical conditioning.Perturbation theory is used to determine sensitivity of problems as well asnumerical stability of algorithms, and the perturbation results built on eachother.For instance, a condition number for matrix multiplication is used to derive aresidual bound for linear systemsolution (Fact 3.5), as well as a least squaresbound for perturbations on the right-hand side (Fact 5.11).No oating point arithmetic.Thereishardlyanymentionofoatingpoint arithmetic, forthreemainreasons. First, sensitivity of numerical problems is, in general, not causedby arithmetic in nite precision. Second, many special-purpose devices inengineering applications perform xed point arithmetic. Third, sensitivityis an issue even in symbolic computation, when input data are not knownexactly.Numerical stability in exact arithmetic.Asimplied concept of numerical stability is introduced to give quantitativeintuition, while avoiding tedious roundoff error analyses. The message isthat unstable algorithms come about if one decomposes a problem into ill-conditioned subproblems.Two bounds for this simpler type of stability are presented for general di-rect solvers (Facts 3.14 and 3.17). These bounds imply, in turn, stabilityboundsforsolversbasedonthefollowingfactorizations: LU(Corollary3.22), Cholesky (Corollary 3.31), and QR (Corollary 3.33).ixSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page xiiiiiiiix PrefaceSimple derivations.The existence of a QRfactorization for nonsingular matrices is deduced verysimply from the existence of a Cholesky factorization (Fact 3.32), withoutany commitment to a particular algorithm such as Householder or GramSchmidt.A new intuitive proof is given for the optimality of the singular value de-composition (Fact 4.13), based on the distance of a matrix from singularity.I derive many relative perturbation bounds with regard to the perturbed so-lution, rather than the exact solution.Such bounds have several advantages:They are computable; they give rise to intermediate absolute bounds (whichareusefulinthecontextofxedpointarithmetic); andtheyareeasytoderive.Especially for full rank least squares problems (Fact 5.14), such a perturba-tion bound can be derived fast, because it avoids the MoorePenrose inverseof the perturbed matrix.High-level view of algorithms.Due to widely available high-quality mathematical software for small densematrices, I believe that it is not necessary anymore to present detailed im-plementations of direct methods in an introductory graduate text. This freesup time for analyzing the accuracy of the output.Complex arithmetic.Results are presented for complex rather than real matrices, because engi-neeringproblemscangiverisetocomplexmatrices. Moreover, limitingones focus to real matrices makes it difcult to switch to complex matriceslater on.Many properties that are often taken for granted in the real case nolonger hold in the complex case.Exercises.The exercises contain many useful facts. Aseparate category of easier exer-cises, labeled with roman numerals, is appropriate for use in class.Acknowledgments. I thank Nick Higham, Rizwana Rehman, Megan Sawyer,and Teresa Selee for providing helpful suggestions and all MA523 students forgivingmetheopportunitytodevelopthematerialinthisbook. Ithasbeenapleasure working with the SIAM publishing staff, in particular with Sara Murphy,who made possible the publication of this book, and Lou Primus, who patientlyand competently dealt with all my typesetting requests.Ilse IpsenRaleigh, NC, USADecember 2008Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page xiiiiiiiiiPreface xiReferencesGene H. Golub and Charles F. Van Loan: Matrix Computations, Third Edition,Johns Hopkins Press, 1996NicholasJ. Higham: AccuracyandStabilityofNumericalAlgorithms, SecondEdition, SIAM, 2002Roger A. Horn and Charles A. Johnson: Matrix Analysis, Cambridge UniversityPress, 1985Peter Lancaster and Miron Tismenetsky: The Theory of Matrices, Second Edition,Academic Press, 1985Carl D. Meyer:Matrix Analysis and Applied Linear Algebra, SIAM, 2000Gilbert Strang: Linear Algebra and Its Applications, Third Edition, Harcourt BraceJovanovich, 1988Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page xiiiiiiiiiiSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page xiiiiiiiiiiiIntroductionThegoalofthisbookistohelpyouunderstandthesensitivityofmatrixcom-putationstoerrorsintheinputdata. Therearetwoimportantreasonsforsucherrors.(i) Input data may not be known exactly.Forinstance, yourweightonthescaletendstobe125pounds, butmaychange to 126 or 124 depending where you stand on the scale. So, you aresure that the leading digits are 12, but you are not sure about the third digit.Therefore the third digit is considered to be in error.(ii) Arithmetic operations can produce errors.Arithmetic operations may not give the exact result when they are carriedout inniteprecision, e.g., inoatingpoint arithmeticorinxedpointarithmetic. This happens, for instance, when 1/3 is computed as .33333333.There are matrix computations that are sensitive to errors in the input.Con-sider the system of linear equations13x1+13x2=1,13x1+.3x2=0,which has the solution x1=27 and x2=30. Suppose we make a small changein the second equation and change the coefcient from .3 to13. The resulting linearsystem13x1+13x2=1,13x1+13x2=0has no solution. Asmall change in the input causes a drastic change in the output,i.e., the total loss of the solution. Why did this happen?How can we predict thatsomething like this can happen? That is the topic of this book.xiiiSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page xiviiiiiiiiSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 1iiiiiiii1. MatricesWe review the basic matrix operations.1.1 What Is a Matrix?An array of numbersA =___a11. . . a1n......am1. . . amn___with m rows and n columns is an mn matrix.Element aijis located in position(i, j).The elements aijare scalars, namely, real or complex numbers.The set ofreal numbers is R, and the set of complex numbers is C.We write ARmnif Ais anmn matrix whose elements are real numbers,and A Cmnif A is an mn matrix whose elements are complex numbers. Ofcourse, RmnCmn.If m=n, then we say that A is a square matrix of order n.For instance,A =_1 2 3 45 6 7 8_is a 24 matrix with elements a13=3 and a24=8.Vectors. A row vectory =_y1. . . ym_ is a 1m matrix, i.e.,y C1m. Acolumn vectorx =___x1...xn___1Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 2iiiiiiii2 1. Matricesis an n1 matrix, i.e., x Cn1or shorter, x Cn. If the elements of x are real,then x Rn.Submatrices. Sometimes we need only those elements of a matrix that are situ-ated in particular rows and columns.Denition 1.1. LetA Cmnhave elementsaij. If 1 i1< i2< < ik mand 1 j1 < j2 0,Ak=k times, .. ,A. . . A =Ak1A =AAk1.Denition 1.3. A square matrix isinvolutory if A2=I,idempotent (or a projector) if A2=A,nilpotent if Ak=0 for some integer k > 0.Example.For any scalar ,_1 0 1_is involutory,_1 0 0_is idempotent,and_0 0 0_is nilpotent.Exercises(i) Which is the only matrix that is both idempotent and involutory?(ii) Which is the only matrix that is both idempotent and nilpotent?(iii) Let x Cn1, y C1n. When is xy idempotent? When is it nilpotent?(iv) Prove:If A is idempotent, then I A is also idempotent.(v) Prove: If Aand B are idempotent and AB =BA, then AB is also idempotent.(vi) Prove: A is involutory if and only if (I A)(I +A) =0.(vii) Prove:If A is involutory and B =12(I +A), then B is idempotent.(viii) Letx Cn1, y C1n. Compute(xy)3xusing only inner products andscalar multiplication.1. Fast Matrix Multiplication.One can multiply two complex numbers with only three real multiplicationsinstead of four. Let =1+2 and =1+2 be two complex numbers,Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 12iiiiiiii12 1. Matriceswhere 2=1 and 1, 2, 1, 2 R. Writing =1122+ [(1+2)(1+2) 1122]shows that the complex product can be computed with three real multi-plications: 11, 22, and (1+1)(2+2).Show that this approach can be extended to the multiplicationABof twocomplex matrices A =A1+A2 and B =B1+B2, where A1, A2 RmnandB1, B2 Rnp. Inparticular, showthat nocommutativitylawsareviolated.1.8 Transpose and Conjugate TransposeTransposing a matrix amounts to turning rows into columns and vice versa.IfA =_____a11a12. . . a1na21a22. . . a2n.........am1am2. . . amn_____,then its transpose AT Cnmis obtained by converting rows to columns,AT=_____a11a21. . . am1a12a22. . . am2.........a1na2n. . . amn_____.Thereis asecondtypeof transpositionthat requires moreworkwhenthematrixelements arecomplexnumbers. Acomplexnumber is written = 1+2, where2= 1and1, 2 R. Thecomplexconjugateofthescalar is =12.If A Cmnis a matrix, its conjugate transpose A Cnmis obtained byconverting rows to columns and, in addition, taking the complex conjugates of theelements,A=_____a11a21. . . am1a12a22. . . am2.........a1na2n. . . amn_____.Example.IfA =_1+2 53 6_,thenAT=_1+2 35 6_, A=_12 3+5 6_.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 13iiiiiiii1.8. Transpose and Conjugate Transpose 13Example. We can express the rows of the identity matrix in terms of canonicalvectors,In=___eT1...eTn___=___e1...en___.Fact 1.4 (Properties of Transposition).For real matrices,the conjugate transpose and the transpose are identical.That is, if A Rmn, then A=AT.Transposing a matrix twice gives back the original,(AT)T=A, (A)=A.Transposition does not affect a scalar, while conjugate transposition conju-gates the scalar,(A)T=AT, (A)=A.The transpose of a sum is the sum of the transposes,(A+B)T=AT+BT, (A+B)=A+B.The transpose of a product is the product of the transposes with the factorsin reverse order,(AB)T=BTAT, (AB)=BA.Example. Why do we have to reverse the order of the factors when the transposeis pulled inside the product AB? Why isnt (AB)T=ATBT?One of the reasonsis that one of the products may not be dened.IfA =_1 11 1_, B =_11_,then(AB)T=_2 2_,while the product ATBTis not be dened.Exercise(i) Let Abe an n nmatrix, andlet Zbe the matrixwith zj,j+1 =1,1 j n1, and all other elements zero. What does ZAZTdo?Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 14iiiiiiii14 1. Matrices1.9 Inner and Outer Products, AgainTransposition comes in handy for the representation of inner and outer products. Ifx =___x1...xn___, y =___y1...yn___,thenxy =x1y1+ +xnyn, yx =y1x1+ +ynxn.Example. Let =1+2 be a complex number, where 2=1 and 1, 2 R.Withx =_12_the absolute value of can be represented as the inner product, || =xx.Fact 1.5 (Properties of Inner Products). Let x, y Cn.1. yx is the complex conjugate of xy, i.e., yx =(xy).2. yTx =xTy.3. xx =0 if and only if x =0.4. If x is real, then xTx =0 if and only if x =0.Proof. Letx =_x1. . . xn_Tandy =_y1. . . yn_T. For the rst equalitywriteyx =

nj=1yjxj =

nj=1xjyj. Since complex conjugating twice givesback the original, we get

nj=1xjyj =

nj=1xjyj =

nj=1xjyj =

nj=1xjyj =xy, where the long overbar denotes complex conjugation over the whole sum.As for the third statement, 0 =xx =

nj=1xjxj =

nj=1|xj|2if and onlyif xj =0, 1 j n, if and only if x =0.Example. The identity matrix can be represented as the outer productIn=e1eT1 +e2eT2 + +eneTn .Exercises(i) Let x be a column vector. Give an example to showthat xTx =0 can happenfor x =0.(ii) Let x Cnand xx =1.Show that In2xx is involutory.(iii) Let A be a square matrix with aj,j+1=1 and all other elements zero. Rep-resent A as a sum of outer products.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 15iiiiiiii1.10. Symmetric and Hermitian Matrices 151.10 Symmetric and Hermitian MatricesWe look at matrices that remain unchanged by transposition.Denition 1.6. A matrix A Cnnissymmetric if AT=A,Hermitian if A=A,skew-symmetric if AT=A,skew-Hermitian if A=A.The identity matrix In is symmetric and Hermitian.The square zero matrix0nn is symmetric, skew-symmetric, Hermitian, and skew-Hermitian.Example.Let 2=1._1 22 4_is symmetric,_1 22 4_is Hermitian,_0 22 0_is skew-symmetric,_1 22 4_is skew-Hermitian.Example.Let 2=1._0 0_is symmetric and skew-Hermitian, while_0 0_is Hermitian and skew-symmetric.Fact 1.7. If A Cmn, then AATand ATA are symmetric, while AA and AAare Hermitian.If A Cnn, then A+ATis symmetric, and A+A is Hermitian.Exercises(i) Is a Hankel matrix symmetric, Hermitian, skew-symmetric, or skew-Hermitian?(ii) Which matrix is both symmetric and skew-symmetric?(iii) Prove:If A is a square matrix, then AATis skew-symmetric and AAis skew-Hermitian.(iv) Which elements of a Hermitian matrix cannot be complex?Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 16iiiiiiii16 1. Matrices(v) What can you say about the diagonal elements of a skew-symmetric matrix?(vi) What can you say about the diagonal elements of a skew-Hermitian matrix?(vii) If A is symmetric and is a scalar, does this imply that A is symmetric?Ifyes, give a proof.If no, give an example.(viii) If A is Hermitian and is a scalar, does this imply that A is Hermitian?Ifyes, give a proof.If no, give an example.(ix) Prove: If Ais skew-symmetric and is a scalar, then Ais skew-symmetric.(x) Prove: If A is skew-Hermitian and is a scalar, then A is, in general, notskew-Hermitian.(xi) Prove:If A is Hermitian, then A is skew-Hermitian, where 2=1.(xii) Prove:If A is skew-Hermitian, then A is Hermitian, where 2=1.(xiii) Prove: If A is a square matrix, then (AA) is Hermitian, where 2=1.1. Prove: Every square matrixA can be writtenA = A1+A2,whereA1isHermitian and A2 is skew-Hermitian.2. Prove:Every square matrix A can be written A =A1+A2, where A1 andA2 are Hermitian and 2=1.1.11 InverseWe want to determine an inverse with respect to matrix multiplication. Inversionof matrices is more complicated than inversion of scalars. There is only one scalarthat does not have an inverse:0.But there are many matrices without inverses.Denition 1.8. A matrixA Cnnis nonsingular (or invertible) ifA has aninverse, that is, if there is a matrix A1so that AA1=I =A1A.If A does nothave an inverse, it is singular.Example.A11 matrix is invertible if it is nonzero.An involutory matrix is its own inverse: A2=I.Fact 1.9. The inverse is unique.Proof. LetA Cnn, and letAB = BA = InandAC = CA = Infor matricesB, C Cnn. ThenB =BIn=B(AC) =(BA)C =InC =C.It is often easier to determine that a matrix is singular than it is to determinethat a matrix is nonsingular. The fact below illustrates this.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 17iiiiiiii1.11. Inverse 17Fact 1.10. Let A Cnnand x, b Cn.If x =0 and Ax =0, then A is singular.If x =0 and A is nonsingular, then Ax =0.If Ax =b, where A is nonsingular and b =0, then x =0.Proof. To prove the rst statement, assume to the contrary thatA is nonsingularand has an inverse A1. Then 0 =Ax implies 0 =A1Ax =Inx =x, hence x =0,which contradicts the assumption x =0. Therefore A must be singular.The proofs for the other two statements are similar.Fact 1.11. An idempotent matrix is either the identity or else is singular.Proof. If A is idempotent, then A2= A. Hence 0 = A2A = A(AI). EitherI A =0, in which case A is the identity, or else I A =0, in which case it hasa nonzero column and Fact 1.10 implies that A is singular.Now we show that inversion and transposition can be exchanged.Fact 1.12. If A is invertible, then ATand A are also invertible, and(A)1=(A1), (AT)1=(A1)T.Proof.Show that (A1) fullls the conditions for an inverse of A:A(A1)=(A1A)=I=Iand(A1)A=(AA1)=I=I.The proof for ATis similar.Because inverse and transpose can be exchanged, we can simply write Aand AT.The expression below is useful because it can break apart the inverse of asum.Fact 1.13 (ShermanMorrison Formula). If A Cnnis nonsingular, andV Cmn, U Cnmare such that I +VA1Uis nonsingular, then(A+UV)1=A1A1U_I +VA1U_1VA1.Here is an explicit expression for the inverse of a partitioned matrix.Fact 1.14. Let A CnnandA =_A11A12A21A22_.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 18iiiiiiii18 1. MatricesIf A11 and A22 are nonsingular, thenA1=_S11A111A12S12A122 A21S11S12_,where S1=A11A12A122 A21 and S2=A22A21A111A12.Matrices of the form S1 and S2 are called Schur complements.Exercises(i) Prove:If A and B are invertible, then (AB)1=B1A1.(ii) Prove: If A, B Cnnare nonsingular, then B1=A1B1(BA)A1.(iii) Let A Cmn, B Cnmbesuchthat I +BAisinvertible. Showthat(I +BA)1=I B(I +AB)1A.(iv) Let A Cnnbe nonsingular, u Cn1, v C1n, and vA1u =1. Showthat(A+uv)1=A1A1uvA11+vA1u.(v) The following expression for the partitioned inverse requires only A11 to benonsingular but not A22.Let A CnnandA =_A11A12A21A22_.Show:If A11 is nonsingular and S =A22A21A111A12, thenA1=_A111 +A111A12S1A21A111A111A12S1S1A21A111S1_.(vi) Let x C1nand A Cnn. Prove: If x =0 and xA=0, then Ais singular.(vii) Prove: The inverse, if it exists, of a Hermitian (symmetric) matrix is alsoHermitian (symmetric).(viii) Prove:If A is involutory, then I A or I +A must be singular.(ix) Let A be a square matrix so that A+A2=I.Prove: A is invertible.(x) Prove: Anilpotent matrix is always singular.1. Let S Rnn.Show:If S is skew-symmetric, then I S is nonsingular.Give an example to illustrate that I S can be singular if S Cnn.2. Let x be a nonzero column vector.Determine a row vector y so that yx =1.3. Let A be a square matrix and letjbe scalars,at least two of which arenonzero, such that

kj=0jAj=0. Prove: If 0=0, then Ais nonsingular.4. Prove: If (I A)1=

kj=0Ajfor some integer k 0, then A is nilpotent.5. Let A, B Cnn. Prove: If I +BAisinvertible, thenI +ABisalsoinvertible.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 19iiiiiiii1.12. Unitary and Orthogonal Matrices 191.12 Unitary and Orthogonal MatricesThese are matrices whose inverse is a transpose.Denition 1.15. A matrix A Cnnis unitary if AA=AA =I, orthogonal if AAT=ATA =I.The identity matrix is orthogonal as well as unitary.Example 1.16. Let c and s be scalars with |c|2+|s|2=1. The matrices_c ss c_,_c ss c_are unitary.The rst matrix above gets its own name.Denition 1.17.If c, s C so that |c|2+|s|2=1, then the unitary 22 matrix_c ss c_iscalledaGivensrotation. If candsarealsoreal, thentheGivensrotation_c ss c_ is orthogonal.When a Givens rotation is real, then both diagonal elements are the same.When a Givens rotation is complex, then the diagonal elements are complex con-jugates of each other. Aunitary matrix of the form_c ss c_,where the real parts of the diagonal elements have different signs, is a reection;it is not a Givens rotation.An orthogonal matrix that can reorder the rows or columns of a matrix iscalled a permutation matrix. It is an identity matrix whose rows have been re-ordered(permuted). Onecanalsothinkofapermutationmatrixasanidentitymatrix whose columns have been reordered.Here is the ofcial denition.Denition 1.18 (Permutation Matrix). A square matrix is a permutation matrixif it contains a single one in each column and in each row, and zeros everywhereelse.Example. The following are permutation matrices.The identity matrix I.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 20iiiiiiii20 1. MatricesThe exchange matrixJ =___1...1___, J_____x1x2...xn_____=_____xn...x2x1_____.The upper circular shift matrixZ =________0 10 1.........11 0________, Z_____x1...xn1xn_____=_____xnx1...xn1_____.Fact 1.19 (Properties of Permutation Matrices).1. Permutation matrices are orthogonal and unitary.That is, if P is a permutation matrix, then PPT=PTP =PP=PP =I.2. The product of permutation matrices is again a permutation matrix.Exercises(i) Prove:If A is unitary, then A, AT, and A are unitary.(ii) What canyousayabout aninvolutorymatrixthat is alsounitary(orthogonal)?(iii) Which idempotent matrix is unitary and orthogonal?(iv) Prove:If A is unitary, so is A, where 2=1.(v) Prove: The product of unitary matrices is unitary.(vi) Partitioned Unitary Matrices.Let A CnnbeunitaryandpartitionA = _A1A2_, whereA1haskcolumns, and A2 has nk columns. Show that A1A1= Ik,A2A2= Ink,and A1A2=0.(vii) Let x Cnand xx =1. Prove: In2xx is Hermitian and unitary. Con-clude that In2xx is involutory.(viii) Show: IfPis a permutation matrix, thenPTandP are also permutationmatrices.(ix) Show:If_P1P2_is a permutation matrix, then_P2P1_is also a permu-tation matrix.1.13 Triangular MatricesTriangular matrices occur frequently during the solution of systems of linear equa-tions, because linear systems with triangular matrices are easy to solve.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 21iiiiiiii1.13. Triangular Matrices 21Denition 1.20. Amatrix A Cnnis upper triangular if aij =0 for i >j. That is,A =___a11. . . a1n......ann___.A matrix A Cnnis lower triangular if ATis upper triangular.Fact 1.21. Let A and B be upper triangular, with diagonal elements ajjand bjj,respectively.A+B and AB are upper triangular.The diagonal elements of AB are aiibii.Ifajj = 0 for allj, thenA is invertible, and the diagonal elements ofA1are 1/ajj.Denition1.22. AtriangularmatrixAisunittriangularifithasonesonthediagonal, and strictly triangular if it has zeros on the diagonal.Example. The identity matrix is unit upper triangular and unit lower triangular.The square zero matrix is strictly lower triangular and strictly upper triangular.Exercises(i) What does an idempotent triangular matrix look like? What does an involu-tory triangular matrix look like?1. Prove: If Ais unit triangular, then Ais invertible, and A1is unit triangular.If A and B are unit triangular, then so is the product AB.2. Show that a strictly triangular matrix is nilpotent.3. Explain why the matrixI eieTjis triangular. When does it have an in-verse?Determine the inverse in those cases where it exists.4. Prove:________1 2. . . n1 ............21 1________1=______1 1 ......1 1______.5. Uniqueness of LU Factorization.Let L1, L2 be unit lower triangular, and U1, U2 nonsingular upper triangular.Prove:If L1U1=L2U2, then L1=L2 and U1=U2.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 22iiiiiiii22 1. Matrices6. Uniqueness of QR Factorization.Let Q1, Q2beunitary(ororthogonal), andR1, R2uppertriangularwithpositive diagonal elements. Prove: IfQ1R1 = Q2R2, thenQ1 = Q2andR1=R2.1.14 Diagonal MatricesDiagonal matrices are special cases of triangular matrices; they are upper and lowertriangular at the same time.Denition 1.23. A matrix A Cnnis diagonal if aij =0 for i =j.That is,A =___a11...ann___.The identity matrix and the square zero matrix are diagonal.Exercises(i) Prove:Diagonal matrices are symmetric. Are they also Hermitian?(ii) Diagonal matrices commute.Prove:If A and B are diagonal, then AB is diagonal, and AB =BA.(iii) Represent a diagonal matrix as a sum of outer products.(iv) Which diagonal matrices are involutory, idempotent, or nilpotent?(v) Prove: If a matrix is unitary and triangular, it must be diagonal. What are itsdiagonal elements?1. Let D be a diagonal matrix.Prove:If D =(I +A)1A, then A is diagonal.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 23iiiiiiii2. Sensitivity, Errors, andNormsTwo difculties arise when we solve systems of linear equations or perform othermatrix computations.(i) Errors in matrix elements.Matrixelementsmaybecontaminatedwitherrorsfrommeasurementsorprevious computations, or they may simply not be known exactly. Merelyinputting numbers into a computer or calculator can cause errors (e.g., when1/3 is stored as .33333333).To account for all these situations, we say thatthematrixelementsareafictedwithuncertaintiesorareperturbed. Ingeneral, perturbations of the inputs cause difculties when the outputs aresensitive to changes in the inputs.(ii) Errors in algorithms.Algorithms may not compute an exact solution, because computing the exactsolution may not be necessary,may take too long,may require too muchstorage, or may not be practical. Furthermore, arithmetic operations in niteprecision may not be performed exactly.In this book, we focus on perturbations of inputs, and howthese perturbationsaffect the outputs.2.1 Sensitivity and ConditioningInreal life, sensitive means1acutelyaffectedbyexternal stimuli, easilyoffendedor emotionally hurt, or responsive to slight changes. Asensitive person can beeasilyupsetbysmallevents, suchashavingtowaitinlineforafewminutes.Hardware can be sensitive: A very slight turn of a faucet may change the waterfrom freezing cold to scalding hot. The slightest turn of the steering wheel whendriving on an icy surface can send the car careening into a spin. Organs can be1The Concise Oxford English Dictionary23Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 24iiiiiiii24 2. Sensitivity, Errors, and Normssensitive: Healthy skin may not even feel the prick of a needle, while it may causeextreme pain on burnt skin.It is no different in mathematics. Steep functions, for instance, can be sen-sitive to small perturbations in the input.Example. Letf (x) = 9xand consider the effect of a small perturbation to theinput of f (50) =950, such asf (50.5) =9950=3f (50).Here a 1 percent change in the input causes a 300 percent change of the output.Systems of linear equations are sensitive when a small modication in thematrix or the right-hand side causes a large change in the solution.Example 2.1. The linear system Ax =b withA =_1/3 1/31/3 .3_, b =_10_has the solutionx =_2730_.However, a small change of the(2, 2) element from .3 to 1/3 results in the totalloss of the solution, because the system Ax =b withA =_1/3 1/31/3 1/3_has no solution.A linear system like the one above whose solution is sensitive to small per-turbationsinthematrixiscalledill-conditioned. Hereisanotherexampleofill-conditioning.Example. The linear system Ax =b withA =_1 11 1+_, b =_11_, 0 1, partition and factorA =_1 n11 an1 a An1_=_1/20a1/2In1__1 00 S__1/21/2a0 In1_,where S An1a1a.3. ComputeS = Ln1Ln1, whereLn1 is lower triangular with positive di-agonal elements.4. ThenL _1/20a1/2Ln1_.ACholesky factorization of a positive matrix is unique.Fact 3.29 (Uniqueness of Cholesky factorization). Let A Cnnbe Hermitianpositive denite. If A =LL where L is lower triangular with positive diagonal,then L is unique.Similarly, if A=LL where L is upper triangular with positivediagonal elements, then L is unique.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 66iiiiiiii66 3. Linear SystemsProof. This can be shown in the same way as the uniqueness of the LU factoriza-tion.Thefollowingresult showsthat onecanuseaCholeskyfactorizationtodetermine whether a Hermitian matrix is positive denite.Fact 3.30. Let A Cnnbe Hermitian. A is positive denite if and only if A =LL where L is triangular with positive diagonal elements.Proof. Algorithm 3.5 shows that ifA is positive denite, thenA = LL. Nowassume that A =LL. Since L is triangular with positive diagonal elements, it isnonsingular. Therefore, Lx =0 for x =0, and xAx =Lx22 > 0.The next bound shows that a Cholesky solver is numerically stable in exactarithmetic.Corollary 3.31 (Stability of Cholesky Solver). LetA Cnnand letA+E beHermitian positive denite matrices, Ax =b, b =0, andA+E =LL, A= E2A2,Ly =b +r1, 1= r1

2b2,Lz =y +r2, 2= r2

2y2.If A1

2E21/2, thenz x2x222(A)(A+1+2(1+1)).Proof. Apply Fact 3.14 to A+E, where S1=L and S2=L. The stability factoris L

2L1

2/(A+E)1

2=1 because Fact 2.19 implies(A+E)1

2=LL1

2=L1

22=L

2L1

2.Exercises(i) The magnitude of an off-diagonal element of a Hermitian positive denitematrixisboundedbythegeometricmeanofthecorrespondingdiagonalelements.Let A Cnnbe Hermitian positive denite. Show:|aij| 0,which implies thatM = AA is Hermitian positive denite. LetM = LLbe aCholesky factorization ofM, whereL is lower triangular with positive diagonalelements. Then M =AA=LL. Multiplying by A on the left gives A=QR,whereQ= AL, and whereR = L is upper triangular with positive diagonalelements.Exercise (ix) in Section 3.6 shows that Q is unitary.The uniqueness of the QR factorization follows from the uniqueness of theCholesky factorization, as well as from Exercise 6 in Section 1.13.Theboundbelowshowsthat aQRsolverisnumericallystableinexactarithmetic.Corollary 3.33 (Stability of QR Solver). Let A Cnnbe nonsingular, Ax =b,b =0, andA+E =QR, A= E2A2,Qy =b +r1, 1= r1

2b2,Rz =y +r2, 2= r2

2y2.If A1

2E21/2, thenz x2x222(A)(A+1+2(1+1)).Proof. Apply Fact 3.14 to A+E, where S1=Q and S2=R. The stability factoris R1

2Q

2/(A+E)1

2=1, because Exercise (v) in Section 2.6 impliesQ

2=1 and (A+E)1

2=R1

2.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 69iiiiiiii3.7. QR Factorization 69There are many ways to compute a QR factorization. Here we present analgorithm that is based on Givens rotations; see Denition 1.17.Givens rotationsareunitary, seeExample1.16, andtheyareoftenusedtointroducezerosintomatrices. Lets start by using a Givens rotation to introduce a single zero into avector.Example.Let x, y C._c ss c__xy_=_d0_, where d =_|x|2+|y|2.Ifx = y = 0, thenc = 1 ands = 0; otherwisec = x/dands = y/d. That is, ifboth components of the vector are zero, then there is nothing to do and the unitarymatrix is the identity.Note that d 0 and |c|2+|s|2=1.When introducing zeros into a longer vector, we embed each Givens rotationin an identity matrix.Example.Suppose we want to zero out elements 2, 3, and 4 in a 41 vector witha unitary matrix. We can apply three Givens rotations in the following order.1. Apply a Givens rotation to rows 3 and 4 to zero out element 4,___1 0 0 00 1 0 00 0 c4s40 0 s4c4______x1x2x3x4___=___x1x2y30___,wherey3 =_|x3|2+|x4|2 0. Ifx4 = x3 = 0, thenc4 = 1ands4 = 0;otherwise c4=x3/y3 and s4=x4/y3.2. Apply a Givens rotation to rows 2 and 3 to zero out element 3,___1 0 0 00 c3s300 s3c300 0 0 1______x1x2y30___=___x1y200___,wherey2 =_|x2|2+|y3|2 0. Ify3 = x2 = 0, thenc3 = 1ands3 = 0;otherwise c3=x2/y2 and s3=y3/y2.3. Apply a Givens rotation to rows 1 and 2 to zero out element 2,___c2s20 0s2c20 00 0 1 00 0 0 1______x1y200___=___y1000___,wherey1 =_|x1|2+|y2|2 0. Ify2 = x1 = 0, thenc2 = 1ands2 = 0;otherwise c2=x1/y1 and s2=y2/y1.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 70iiiiiiii70 3. Linear SystemsTherefore Qx =y1e1, where y1=Qx2 andQ=___c2s20 0s2c20 00 0 1 00 0 0 1______1 0 0 00 c3s300 s3c300 0 0 1______1 0 0 00 1 0 00 0 c4s40 0 s4c4___.TherearemanypossibleordersinwhichtoapplyGivensrotations, andGivensrotationsdont havetooperateonadjacent rowseither. Theexamplebelow illustrates this.Example. Here is another way to to zero out elements 2, 3, and 4 in a 41 vector.We can apply three Givens rotations that all involve the leading row.1. Apply a Givens rotation to rows 1 and 4 to zero out element 4,___c40 0 s40 1 0 00 0 1 0s40 0 c4______x1x2x3x4___=___y1x2x30___,wherey1 =_|x1|2+|x4|2 0. Ifx4 = x1 = 0, thenc4 = 1ands4 = 0;otherwise c4=x1/y1 and s4=x4/y1.2. Apply a Givens rotation to rows 1 and 3 to zero out element 3,___c30 s300 1 0 0s30 c300 0 0 1______y1x2x30___=___z1x200___,wherez1 =_|y1|2+|x3|2 0. Ifx3 = y1 = 0, thenc3 = 1ands3 = 0;otherwise c3=y1/z1 and s3=x3/z1.3. Apply a Givens rotation to rows 1 and 2 to zero out element 2,___c2s20 0s2c20 00 0 1 00 0 0 1______x1y200___=___u1000___,whereu1 =_|z1|2+|x2|2 0. Ifx2 = z1 = 0, thenc2 = 1ands2 = 0;otherwise c2=z1/u1 and s2=x2/u1.Therefore Qx =u1e1, where u1=Qx2 andQ=___c2s20 0s2c20 00 0 1 00 0 0 1______c30 s300 1 0 0s30 c300 0 0 1______c40 0 s40 1 0 00 0 1 0s40 0 c4___.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 71iiiiiiii3.7. QR Factorization 71The preceding examples demonstrate that if a Givens rotation operates onrowsiandj, then thec andselements occupy positions(i, i),(i, j),(j, i), and(j, j).Atlasthereisasketchofhowonecanreduceasquarematrixtouppertriangular form by means of Givens rotations.Example. We introduce zeros one column at a time, from left to right, and withinacolumnfrombottomtotop. TheGivensrotationsoperateonadjacentrows.Elements that can be nonzero are represented by . Elements that were affectedbytheithGivensrotationhavethelabel i. Westartbyintroducingzerosintocolumn 1,___ ___1___ 1 1 1 10 1 1 1___2___ 2 2 2 20 2 2 20 1 1 1___3___3 3 3 30 3 3 30 2 2 20 1 1 1___.Now we introduce zeros into column 2, and then into column 3,___3 3 3 30 3 3 30 2 2 20 1 1 1___4___3 3 3 30 3 3 30 4 4 40 0 4 4___5___3 3 3 30 5 5 50 0 5 50 0 4 4___6___3 3 3 30 5 5 50 0 6 60 0 0 6___.Below is the general algorithm.Algorithm 3.6. QR Factorization for Nonsingular Matrices.Input: Nonsingular matrix A CnnOutput: Unitary matrix Q Cnnand upper triangular matrix R Cnnwith positive diagonal elements such that A =QR1. If n =1, then QA/|A| and R |A|.2. If n > 1, zero out elements n, n1, . . . , 2 in column 1 of A as follows.(i) Set _bn1bn2. . . bnn_=_an1an2. . . ann_.(ii) For i =n, n1, . . . , 2Zero out element (i, 1) by applying a rotation to rows i and i 1,_cisisici__ai1,1ai1,2. . . ai1,nbi1bi2. . . bin_=_bi1,1bi1,2. . . bi1,n0 ai2. . . ain_,wherebi1,1 _|bi1|2+|ai1,1|2. Ifbi1 = ai1,1 = 0,thenci 1and si 0; otherwise ci ai1,1/bi1,1 and si bi1/bi1,1.(iii) Multiply all n1 rotations,Qn__c2s20s2c200 0 In2__ __In20 00 cnsn0 sncn__.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 72iiiiiiii72 3. Linear Systems(iv) Partition the transformed matrix,QnA =_r11r0A_, whereA ___ a22. . . a2n...... an2. . . ann___,r_b12. . . b1n_, and r11b11 > 0.3. ComputeA = Qn1Rn1, whereQn1 is unitary andRn1 is upper trian-gular with positive diagonal elements.4. ThenQQn_1 00 Qn1_, R _r11r0 Rn1_.Exercises(i) Determine the QR factorization of a real upper triangular matrix.(ii) QR Factorization of Outer Product.Let x, y Cn, and apply Algorithm 3.6 to xy.How many Givens rotationsdo you have to apply at the most?What does the upper triangular matrix Rlook like?(iii) Let A Cnnbe a tridiagonal matrix, that is, only elements aii, ai+1,i, andai,i+1can be nonzero; all other elements are zero. We want to compute aQR factorizationA = QR withn1 Givens rotations. In which order dothe elements have to be zeroed out, on which rows do the rotations act, andwhich elements of R can be nonzero?(iv) QL Factorization.Show: Everynonsingular matrixA Cnnhas auniquefactorizationA = QL, whereQisunitaryandLislowertriangularwithpositivedi-agonal elements.(v) Computation of QL Factorization.Suppose we want to compute the QL factorization of a nonsingular matrixA Cnnwith Givens rotations.In which order do the elements have to bezeroed out, and on which rows do the rotations act?(vi) The elements in a Givens rotationG=_c ss c_are named to invoke an association with sine and cosine, because|c|2+|s|2= 1. Onecanalsoexpresstheelementsintermsoftangentsor cotangents.LetG_xy_=_d0_, where d =_|x|2+|y|2.Show the following:If |y| >|x|, then =xy, s =y|y|1_1+||2, c =s,Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 73iiiiiiii3.8. QR Factorization of Tall and Skinny Matrices 73and if |x| >|y|, then =yx, c =x|x|1_1+||2, s =c.(vii) Householder Reections.Hereisanother waytointroducezerosintoavector without changingits two norm. Letx Cnandx1 = 0. DeneQ = I 2vv/vv, wherev =x +x2e1and =x1/|x1|. Showthat Qis unitaryandthatQx =x2e1. The matrix Q is called a Householder reection.(viii) Householder Reections for Real Vectors.Let x, y Rnwith x2 = y2. ShowhowtochooseavectorvintheHouseholder reection so that Qx =y.3.8 QR Factorization of Tall and Skinny MatricesWe look at rectangular matrices A Cmnwith at least as many rows as columns,i.e., mn.If A is involved in a linear system Ax =b, then we must have b Cmandx Cn. Such linear systems do not always have a solution;and if they dohappen to have a solution, then the solution may not be unique.Example.IfA =__111__, b =__b1b2b3__,then the linear systemAx =b has a solution only for those b all of whose elementsare the same, i.e., =b1=b2=b3.In this case the solution is x =.Fortunately, there is one right-hand side for which a linear systemAx = balways has a solution,namely, b = 0. That is, Ax = 0 always has the solutionx =0.However, x =0 may not be the only solution.Example.IfA =__1 11 11 1__,then Ax =0 has innitely many solutions x =_x1x2_Twith x1=x2.We distinguish matrices A where x =0 is the unique solution for Ax =0.Denition 3.34. Let A Cmn. The columns of A are linearly independent ifAx = 0 implies x = 0. If Ax = 0 has innitely many solutions, then the columnsof A are linearly dependent.Example. The columns of a nonsingular matrix A are linearly independent. If A is nonsingular, then the matrix_A0_has linearly independent columns.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 74iiiiiiii74 3. Linear Systems Let x Cn. If x = 0, thenxconsistsofasingle, linearlyindependentcolumn.If x =0, then x is linearly dependent. If A Cmnwith AA=In, then Ahas linearly independent columns. Thisis because multiplying Ax =0 on the left by A implies x =0. If the linear systemAx = b has a solutionx, then the matrixB =_A b_has linearly dependent columns. That is because B_x1_=0.How can we tell whether a tall and skinny matrix has linearly independentcolumns? We can use a QR factorization.Algorithm 3.7. QR Factorization for Tall and Skinny Matrices.Input: Matrix A Cmnwith mnOutput: Unitary matrix Q Cmmand upper triangular matrix R Cnnwith nonnegative diagonal elements such that A =Q_R0_1. Ifn = 1, thenQ is a unitary matrix that zeros out elements 2, . . . , m ofA,and R A2.2. If n > 1, then, as in Algorithm 3.6, determine a unitary matrix Qm Cmmto zero out elements 2, . . . , m in column 1 of A, so thatQmA =_r11r0A_,where r110 and A C(m1)(n1).3. Compute A=Qm1_Rn10_, where Qm1 C(m1)(m1)is unitary, andRn1C(n1)(n1)is upper triangular withnonnegative diagonal elements.4. ThenQQm_1 00 Qm1_, R _r11r0 Rn1_.Fact 3.35. Let A Cmnwith mn, and A =Q_R0_where Q Cmmis uni-tary, and R Cnnis upper triangular. Then A has linearly independent columnsif and only if R has nonzero diagonal elements.Proof. Since Qis nonsingular, Ax =Q_R0_0 x =0 if and only ifRx =0 x =0. This is the case if and only if R is nonsingular and has nonzerodiagonal elements.One can make a QR factorization more economical by reducing the storageand omitting part of the unitary matrix.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 75iiiiiiii3.8. QR Factorization of Tall and Skinny Matrices 75Fact 3.36 (Thin QR Factorization). IfA Cmnwithm n, then there existsa matrix Q1 Cmnwith Q1Q1= In, and an upper triangular matrix R Cnnwith nonnegative diagonal elements so that A =Q1R.Proof. Let A =Q_R0_be a QRfactorizationas inFact 3.35. PartitionQ=_Q1Q2_, where Q1 has n columns. Then A =Q1R.Denition3.37. If ACmnandAA=In, thenthe columns of Aare orthonormal.For a square matrix the thin QR decomposition is identical to the full QRdecomposition.Example 3.38. The columns of a unitary or an orthogonal matrixA CnnareorthonormalbecauseAA = In, andsoaretherowsbecauseAA = In. Thismeans, a square matrix with orthonormal columns must be a unitary matrix. Arealsquare matrix with orthonormal columns is an orthogonal matrix.Exercises(i) Let A Cmn, m n, withthinQRfactorization A =QR. Show:A2=R2.(ii) Uniqueness of Thin QR Factorization.Let A Cmnhavelinearlyindependent columns. Show: If A = QR,where Q Cmnsatises QQ=In and R is upper triangular with positivediagonal elements, then Q and R are unique.(iii) Generalization of Fact 3.35.Let A Cmn, m n, andA = B_C0_, whereB Cmnhaslinearlyindependent columns,andC Cnn. Show: A has linearly independentcolumns if and only if C is nonsingular.(iv) LetA Cmnwherem > n. Show: There exists a matrixZ Cm(mn)such that ZA =0.(v) LetA Cmn,m n, have a thin QR factorization A = QR. Express thekth column of Aas a linear combination of columns of Qand elements of R.How many columns of Q are involved?(vi) Let A Cmn, m n, have a thin QR factorization A =QR. Determine aQR factorization of AQe1e1R from the QR factorization of A.(vii) LetA =_a1. . . an_ have linearly independent columnsaj, 1 j n.Let A =QR be a thin QR factorization where Q=_q1. . . qn_ and R isupper triangular with positive diagonal elements. Express the elements of Rin terms of the columns ajof A and the columns qjof Q.(viii) LetA be a matrix with linearly independent columns. Show how to com-putethelower-upperCholeskyfactorizationofAAwithoutformingtheproduct AA.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 76iiiiiiii76 3. Linear Systems(ix) Bessels Inequality.Let V Cmnwith V =_v1. . . vn_ have orthonormal columns, and letx Cm.Show:n

j=1|vjx|2xx.1. QR Factorization with Column Pivoting.This problem presents a method to compute QR factorizations of arbitrarymatrices.Let A Cmnwith rank(A) =r. Then there exists a permutationmatrix Pso thatAP =Q_R1R20 0_,where R1 is an upper triangular nonsingular matrix.(a) Show how to modify Algorithm 3.7 so that it computes such a factor-ization. In the rst step, choose a permutation matrixPnthat bringsthe column with largest two norm to the front; i.e.,APne1

2=max1jnAPnej

2.(b) Show that the diagonal elements ofR1have decreasing magnitudes;i.e., (R1)11(R1)22 (R1)rr.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 77iiiiiiii4. Singular ValueDecompositionIn order to solve linear systems with a general rectangular coefcient matrix, weintroduce the singular value decomposition. It is one of the most important toolsin numerical linear algebra, because it contains a lot of information about a matrix,including rank, distance to singularity, column space, row space, and null spaces.Denition4.1(SVD). Let ACmn. If mn, thenasingular value decomposition(SVD) of A is a decompositionA =U_

0_V, where=___1...n___, 1 n0,and U Cmmand V Cnnare unitary.If mn, then an SVD of A isA =U_ 0_V, where=___1...m___, 1 m0,and U Cmmand V Cnnare unitary.ThematrixUiscalledaleftsingularvectormatrix, Viscalledarightsingular vector matrix, and the scalars jare called singular values.Remark 4.2.An mn matrix has min{m, n} singular values.The singular values are unique, but the singular vector matrices are not.Although an SVD is not unique, one often says the SVD instead of anSVD.77Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 78iiiiiiii78 4. Singular Value DecompositionLet A Cmnwithm n. If A =U_

0_Vis anSVDof A, thenA= V_ 0_Uis an SVD ofA. Therefore,A andAhave the samesingular values.A Cnnis nonsingular if and only if all singular values are nonzero, i.e.,j> 0, 1 j n.If A =UV is an SVD of A, then A1=V1U is an SVD of A1.Example 4.3. The 22 matrixA =_1 0 1_has a smallest singular value equal to2=_22+||2+||_4+||2_1/2.As || , thesmallestsingularvalueapproacheszero, 2 0, sothattheabsolute distance of A to singularity decreases.Exercises(i) Let A Cnn.Show: All singular values of A are the same if and only if Ais a multiple of a unitary matrix.(ii) Show that the singular values of a Hermitian idempotent matrix are 0 and 1.(iii) Show: A Cnnis Hermitian positive denite if and only if it has an SVDA =VV whereis nonsingular.(iv) Let A, B Cmn. Show:Aand B have the same singular values if and onlyif there exist unitary matrices QCnnand P Cmmsuch that B =PAQ.(v) Let A Cmn, m n, with QRdecomposition A =Q_R0_, whereQ CmmisunitaryandR Cnn. DetermineanSVDof AfromanSVD of R.(vi) Determine an SVD of a column vector, and an SVD of a row vector.(vii) Let A Cmnwithm n. Show: Thesingularvaluesof AAarethesquares of the singular values of A.1. Show: If ACnnis Hermitian positive denite and >n, then A+Inis also Hermitian positive denite with singular values j+.2. Let A Cmnand > 0. Express the singular values of (AA+I)1Ain terms of and the singular values of A.3. Let A Cmnwith mn.Show: The singular values of_InA_are equal to_1+2j , 1 j n.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 79iiiiiiii4.1. Extreme Singular Values 794.1 Extreme Singular ValuesThe smallest and largest singular values of a matrix provide information about thetwo normof the matrix, the distance to singularity, and the two normof the inverse.Fact 4.4 (Extreme Singular Values). If ACmnhas singular values1 p, where p =min{m, n}, thenA2=maxx=0Ax2x2=1, minx=0Ax2x2=p.Proof. Thetwonormof Adoesnot changewhenAismultipliedbyunitarymatrices; see Exercise (iv) in Section 2.6. Hence A2 = 2. Sinceis adiagonal matrix, Exercise (i) in Section 2.6 implies 2=maxj|j| =1.Toshowtheexpressionfor p, assumethat m n, sop = n. ThenAhas an SVDA = U_

0_V. Letz be a vector so that z2 = 1 and Az2 =minx2=1Ax2. With y =Vz we getminx2=1Ax2=Az2=Vz2=y2=_n

i=12i |yi|2_1/2ny2=n.Thus, nminx2=1Ax2. As for the reverse inequality,n=en

2=UAVen

2=A(Ven)2 minx2=1Ax2.The proof for m < n is analogous.Theextremesingularvaluesareusefulbecausetheyprovideinformationaboutthetwo-normconditionnumberwithrespecttoinversion, andaboutthedistance to singularity.The expressions below show that the largest singular value determines howmuchamatrixcanstretchaunit-normvectorandthesmallest singularvaluedetermines how much a matrix can shrink a unit-norm vector.Fact 4.5. If A Cnnis nonsingular with singular values 1 n >0, thenA1

2=1n, 2(A) =1n.The absolute distance of A to singularity isn=min{E2: A+E is singular}and the relative distance isn1=min_E2A2: A+E is singular_.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 80iiiiiiii80 4. Singular Value DecompositionProof. Remark4.2impliesthat 1/jarethesingularvaluesof A1, sothatA1

2 = maxj 1/|j| = 1/n. The expressions for the distance to singularityfollow from Fact 2.29 and Corollary 2.30.Fact 4.5 implies that a nonsingular matrix is almost singular in the absolutesenseifitssmallestsingularvalueisclosetozero. Ifthesmallestandlargestsingular values are far apart, i.e., if 1n, then the matrix is ill-conditioned withrespect to inversion in the normwise relative sense, and it is almost singular in therelative sense.The singular values themselves are well-conditioned in the normwise abso-lute sense. We show this below for the extreme singular values.Fact 4.6. Let A, A+E Cmn, p = min{m, n}, andlet 1 pbethesingular values of A and 1 p the singular values of A+E. Then| 11| E2, | pp| E2.Proof.The inequality for 1 follows from 1=A2 and Fact 2.13, which statesthat norms are well-conditioned.Regardingtheboundfor p, let ybeavectorsothat p = Ay2andy2=1. Then the triangle inequality implies p=minx2=1(A+E)x2(A+E)y2Ay2+Ey2=p+Ey2p+E2.Hence ppE2. To show that E2 pp, let y be a vector so that p=(A+E)y2 and y2=1. Then the triangle inequality yieldsp=minx2=1Ax2Ay2=(A+E)y Ey2(A+E)y2+Ey2= p+Ey2 p+E2.Exercises1. Extreme Singular Values of a Product.Let A Ckm, B Cmn, q =min{k, n}, and p =min{m, n}.Show:1(AB) 1(A)1(B), q(AB) 1(A)p(B).2. Appending a column to a tall and skinny matrix does not increase the smallestsingularvaluebutcandecreaseit, becausethenewcolumnmaydependlinearly on the old ones. The largest singular value does not decrease but itcan increase, because more mass is added to the matrix.Let A Cmnwith m>n, z Cm, and B = _A z_. Show:n+1(B) n(A) and 1(B) 1(A).3. Appending a row to a tall and skinny matrix does not decrease the smallestsingular value but can increase it. Intuitively, this is because the columnsSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 81iiiiiiii4.2. Rank 81become longer which gives them an opportunity to become more linearlyindependent.The largest singular value does not decrease but can increase,because more mass is added to the matrix.Let A Cmnwith mn, z Cn, and B =_Az_.Show thatn(B) n(A), 1(A) 1(B) _1(A)2+z22.4.2 RankFor a nonsingular matrix, all singular values are nonzero. For a general matrix, thenumber of nonzero singular values measures howmuch information is containedinamatrix, whilethenumberofzerosingularvaluesindicatestheamountofredundancy.Denition4.7(Rank). Thenumber of nonzerosingular values of amatrixA Cmnis called the rank of A. An mn zero matrix has rank 0.Example 4.8.If A Cmn, then rank(A) min{m, n}.This follows from Remark 4.2.If A Cnnis nonsingular, then rank(A) =n =rank(A1).A nonsingular matrix A contains the maximum amount of information, be-cause it can reproduce any vector b Cnby means of b =Ax.For any mn zero matrix 0, rank(0) =0.Thezeromatrixcontainsnoinformation. Itcanonlyreproducethezerovector, because 0x =0 for any vector x.If A Cmnhas rank(A) = n, then A has an SVD A =U_

0_V, where is nonsingular. This means, all singular values of A are nonzero.If ACmnhas rank(A) =m, then Ahas an SVDA=U_ 0_V, where is nonsingular. This means, all singular values of A are nonzero.A nonzero outer product uvcontains little information: becauseuvx =(vx)u, the outer product uv can produce only multiples of the vector u.Remark 4.9 (Outer Product). If u Cmand v Cnwith u =0 and v =0, thenrank(uv) =1.To see this, determine an SVD of uv.Let U Cmmbe a unitary matrix sothat Uu =u2e1, and let V Cnnbe a unitary matrix so that Vv =v2e1.Substituting these expressions into uv shows that uv=UV is an SVD, where Rmnand = u2v2e1e1. Therefore, thesingularvaluesof uvareu2v2, and (min{m, n} 1) zeros.In particular, uv

2=u2v2.The above example demonstrates that a nonzero outer product has rank one.Nowweshowthatamatrixofrankr canberepresentedasasumof r outerproducts. To this end we distinguish the columns of the left and right singularvector matrices.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 82iiiiiiii82 4. Singular Value DecompositionDenition 4.10 (Singular Vectors). Let A Cmn, with SVD A =U_

0_V ifmn, and SVD A =U_ 0_V if mn.Set p =min{m, n} and partitionU =_u1. . . um_, V =_v1. . . vn_,=___1...p___,where 1 p0.We call jthe jth singular value, ujthe jth left singular vector, and vjthejth right singular vector.Corresponding left and right singular vectors are related to each other.Remark 4.11.Let A have an SVD as in Denition 4.10.ThenAvi =iui, Aui =ivi, 1 i p.This follows from the fact that Uand Vare unitary, andis Hermitian.Nowwe are ready to derive an economical representation for a matrix, wherethe size of the representation is proportional to the rank of the matrix. Fact 4.12below shows that a matrix of rank r can be expressed in terms of r outer products.Theseouterproductsinvolvethesingularvectorsassociatedwiththenonzerosingular values.Fact 4.12 (Reduced SVD). Let A Cmnhave an SVD as in Denition 4.10.Ifrank(A) =r, thenA =r

j=1jujvj.Proof. Fromrank(A) =r follows 1 r>0. Conne the nonzero singularvalues to the matrix r, so that

r =___1...r___, and A =U_

r00 0_Vis an SVD ofA. Partitioning the singular vectors conformally with the nonzerosingular values,U = _r mrUrUmr_, V = _r nrVrVnr_,yieldsA = Ur

rVr . UsingUr =_u1. . . ur_andVr =_v1. . . vr_, andviewing matrix multiplication as an outer product,as in View 4 of Section 1.7,showsA =Ur

rVr =_1u1. . . rur____v1...vr___=r

j=1jujvj.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 83iiiiiiii4.2. Rank 83For a nonsingular matrix, the reduced SVD is equal to the ordinary SVD.Based on the above outer product representation of a matrix, we will nowshow that the singular vectors associated with thek largest singular values ofAdetermine the rankk matrix that is closest toA in the two norm. Moreover, the(k +1)st singular value of A is the absolute distance of A, in the two norm, to theset of rank k matrices.Fact 4.13 (Optimality of the SVD). Let A CmnhaveanSVDasinDeni-tion4.9. If k< rank(A), thentheabsolutedistanceof Atothesetofrankkmatrices isk+1= minBCmn,rank(B)=kAB2=AAk

2,where Ak =

kj=1jujvj.Proof. Write the SVD asA =U_

1

2_V, where 1=___1...k+1___and1 k+1> 0, so that1 is nonsingular. The idea is to show that thedistance of 1 to the set of singular matrices, which is k+1, is a lower bound forthe distance of A to the set of all rank k matrices.Let C Cmnbe a matrix with rank(C) =k, and partitionUCV =_C11C12C21C22_,where C11 is (k +1)(k +1). Fromrank(C) =k follows rank(C11) k (althoughit is intuitively clear, it is proved rigorously in Fact 6.19), so thatC11 is singular.Since the two normis invariant under multiplication by unitary matrices, we obtainAC2=_____

1

2_UCV____2=_____

1C11C12C21

2C22_____21C11

2.Since 1is nonsingular and C11is singular, Facts 2.29and4.5implythat

1C11

2 is bounded below by the distance of 1 from singularity, and

1C11

2min{1B11

2: B11 is singular} =k+1.Amatrix C for which AC2=k+1 is C =Ak. This is becauseC11=_____1...k0_____, C12=0, C21=0, C22=0.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 84iiiiiiii84 4. Singular Value DecompositionSince the 1C11 has k diagonal elements equal to zero, and the diagonal elementsof 2 are less than or equal to k+1, we obtainAC2=_____

1C1100 2_____2=_____k+1

2_____2=k+1.The singular values also help us to relate the rank ofA to the rank ofAAand AA. This will be important later on for the solution of least squares problems.Fact 4.14. For any matrix A Cmn,1. rank(A) =rank(A),2. rank(A) =rank(AA) =rank(AA),3. rank(A) =n if and only if AA is nonsingular,4. rank(A) =m if and only if AA is nonsingular.Proof.1. This follows from Remark 4.2, becauseA andAhave the same singularvalues.2. Ifm n,thenA has an SVDA =U_

0_V,andAA = V2Vis anSVD of AA. Sinceand 2have the same number of nonzero diagonalelements, rank(A) = rank(AA). Also,AA= U_

200 0_U is an SVDof AA. As before, rank(A) =rank(AA) becauseand 2have the samenumber of nonzero diagonal elements.Asimilar argument applies when m < n.3. SinceAA isn n, AA is nonsingular if and only ifn = rank(AA) =rank(A), where the second equality follows from item 2.4. The proof is similar to that of item 3.In item 3 above the matrix A has linearly independent columns, and in item4 it has linearly independent rows.Below we give another name to such matrices.Denition4.15(Full Rank). Amatrix A Cmnhas full columnrankifrank(A) =n, and full row rank if rank(A) =m.A matrix A Cmnhas full rank if A has full column rank or full row rank.A matrix that does not have full rank is rank decient.Example. Anonsingular matrix has full row rank and full column rank. A nonzero column vector has full column rank, and a nonzero row vectorhas full row rank. IfA Cnnis nonsingular, then _A B_ has full row rank for any matrixB Cnm, and_AC_ has full column rank, for any matrix C Cmn. Asingular square matrix is rank decient.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 85iiiiiiii4.2. Rank 85Below we show that matrices with orthonormal columns also have full col-umn rank. Recall fromDenition 3.37 that Ahas orthonormal columns if AA=I.Fact 4.16. A matrix A Cmnwith orthonormal columns has rank(A) = n, andall singular values are equal to one.Proof. Fact 4.14implies rank(A) =rank(AA) =rank(In) =n. Thus Ahas full columnrank, andwecanwriteits SVDas A =U_

0_V. ThenIn = AA = V2Vimplies = In, so that all singular values ofA are equalto one.Exercises(i) Let A Cmn. Show: If Q CmmandP Cnnareunitary, thenrank(A) =rank(QAP).(ii) What can you say about the rank of a nilpotent matrix, and the rank of anidempotent matrix?(iii) Let A Cmn. Show:If rank(A) =n, then (AA)1A

2=1/n, and ifrank(A) =m, then (AA)1A2=1/m.(iv) LetA Cmnwith rank(A) = n. Show thatA(AA)1Ais idempotentand Hermitian, and A(AA)1A

2=1.(v) LetA Cmnwith rank(A) = m. Show thatA(AA)1A is idempotentand Hermitian, and A(AA)1A2=1.(vi) Nilpotent Matrices.LetA Cnnbe nilpotent so thatAj= 0 andAj1= 0 for somej 1.Let b Cnwith Aj1b =0. Show that K =_b Ab . . . Aj1b_has fullcolumn rank.(vii) In Fact 4.13 let B be a multiple of Ak, i.e., B =Ak.Determine AB2.1. Let A Cnn. Showthat there exists a unitarymatrix QsuchthatA=QAQ.2. Polar Decomposition.Show: If A Cmnhas rank(A) =n, then there is a factorization A=PH,whereP Cmnhasorthonormalcolumns, andH CnnisHermitianpositive denite.3. The polar factor Pis the closest matrix with orthonormal columns in the twonorm.Let A Cnnhave a polar decomposition A =PH. ShowthatAP2AQ2 for any unitary matrix Q.4. The distance of a matrixA from its polar factorPis determined by howclose the columns A are to being orthonormal.Let A Cmn, with rank(A) = n, have a polar decompositionA = PH.Show thatAAIn

21+A2AP2 AAIn

21+n.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 86iiiiiiii86 4. Singular Value Decomposition5. LetA Cnnand> 0. Show: is a singular value ofA if and only ifthe matrix_A II A_is singular.6. Rank Revealing QR Factorization.Withanappropriatepermutationofthecolumns, aQRfactorizationcanalmost reveal the smallest singular value of a full column rank matrix.Let A Cmnwith rank(A) =n and smallest singular value n. Let the cor-responding singular vectors be Av =nu, where v2=u2=1.Choosea permutation Pso that w =Pv and |wn| =w, and let AP =Q_R0_be a QR decomposition of AP.Show: |rnn| nn.4.3 Singular VectorsThe singular vectors of a matrix A give information about the column spaces andnull spaces of A and A.The column space of a matrix A is the set of all right-hand sides b for whichthe systemAx =b has a solution, and the null space of Adetermines whether thesesolutions are unique.Denition 4.17 (Column Space and Null Space).If A Cmn, then the setR(A) ={b Cm: b =Ax for some x Cn}is the column space or range of A, and the setKer(A) ={x Cn: Ax =0}is the kernel or null space of A.Example. The column space of anmn zero matrix is the zero vector, and the nullspace is Cn, i.e., R(0mn) ={0m1} and Ker(0mn) =Cn. The column space of annn nonsingular complex matrix is Cn, and thenull space consists of the single vector 0n1. Ker(A) = {0} if and only if the columns of the matrixA are linearly inde-pendent. If A Cmn, then for all k 1R_A 0mk_=R(A), Ker_A0kn_=Ker(A).Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 87iiiiiiii4.3. Singular Vectors 87 If A Cnnis nonsingular, then for any B Cnpand C CpnR_A B_=R(A), Ker_AC_={0n1}.ThecolumnandnullspacesofAarealsoimportant, andwegivethemnames that relate to the matrix A.Denition 4.18 (Row Space and Left Null Space).Let A Cmn.The setR(A) ={d Cn: d =Ay for some y Cm}is the row space of A.The setKer(A) ={y Cm: Ay =0}is the left null space of A.Note that all spaces of a matrix are dened by column vectors.Example 4.19. If Ais Hermitian, thenR(A) =R(A) andKer(A) =Ker(A).The singular vectors reproduce the four spaces associated with a matrix. LetA Cmnwith rank(A) =r and SVDA =U_

r00 0_V,where ris nonsingular, and_r nrr r0mr 0 0_, U = _r mrUrUmr_, V = _r nrVrVnr_.Fact 4.20 (Spaces of a Matrix and Singular Vectors). Let A Cmn.1. The leading r left singular vectors represent the column space of A:If A =0, then R(Ur) =R(A); otherwise R(A) ={0m1}.2. The trailing nr right singular vectors represent the null space of A:If rank(A) =r< n, then R(Vnr) =Ker(A); otherwise Ker(A) ={0n1}.3. The leading r right singular vectors represent the row space of A:If A =0, then R(A) =R(Vr); otherwise R(A) ={0n1}.4. The trailing mr left singular vectors represent the left null space of A:If r< m, then R(Umr) =Ker(A); otherwise Ker(A) ={0m1}.Proof. Although the statements may be intuitively obvious, they are proved rigor-ously in Section 6.1.The singular vectors help us to relate the spaces ofAA andAA to thoseof the matrix A.Since AA and AA are Hermitian, we need to specify only twospaces; see Example 4.19.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 88iiiiiiii88 4. Singular Value DecompositionFact 4.21 (Spaces of AAand AA). Let A Cmn.1. Ker(AA) =Ker(A) and R(AA) =R(A).2. R(AA) =R(A) and Ker(AA) =Ker(A).Proof.Fact 4.14 implies that AA and AA have the same rank as A.Since AAhas the same right singular vectors asA, Fact 4.20 implies Ker(AA) = Ker(A)and R(AA) =R(A). SinceAA has the same left singular vectors asA, Fact4.20 implies R(AA) =R(A) and Ker(AA) =Ker(A).In the special case when the rank of a matrix is equal to the number of rows,then the number of elements in the column space is as large as possible. When therank of the matrix is equal to the number of columns, then the number of elementsin the null space is as small as possible.Fact 4.22 (Spaces of Full Rank Matrices). Let A Cmn. Then1. rank(A) =m if and only if R(A) =Cm;2. rank(A) =n if and only if Ker(A) ={0}.Proof.Let A =U_

r00 0_V be an SVD of A, where ris nonsingular.1. FromFact 4.20 followsR(A) =R(Ur). Hence r =mif and only if Ur =U,because Uis nonsingular so that R(U) =Cm.2. Fact 4.20 also implies r =n if and only if Vnris empty, which means thatKer(A) ={0}.If the matrix in a linear system has full rank, then existence or uniqueness ofa solution is guaranteed.Fact 4.23 (Solutions of Full Rank Linear Systems). Let A Cmn.1. Ifrank(A) = m, thenAx = bhasasolutionx = A(AA)1bforeveryb Cm.2. If rank(A) =nandif b R(A), thenAx =bhastheuniquesolutionx =(AA)1Ab.Proof.1. Fact 4.22implies that Ax =b has a solutionfor every b Cm, andFact 4.14 implies that AA is nonsingular.Clearly, x =A(AA)1b satis-es Ax =b.2. Sinceb R(A), Ax = bhasasolution. Multiplyingontheleft byAgivesAAx = Ab. According to Fact 4.14, AA is nonsingular,so thatx =(AA)1Ab.SupposeAx = bandAy = b; thenA(x y) = 0. Fact 4.22 implies thatKer(A) ={0}, so x =y, which proves uniqueness.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 89iiiiiiii4.3. Singular Vectors 89Exercises(i) Fredholms Alternatives.(a) The rst alternative implies thatR(A) and Ker(A) have only the zerovector in common. Assume b =0 and show:If Ax =b has a solution, then bA =0.In other words, if b R(A), then b Ker(A).(b) The second alternative implies that Ker(A) and R(A) have only thezero vector in common. Assume x =0 and show:If Ax =0, then there is no y such that x =Ay.In other words, if x Ker(A), then x R(A),(ii) Normal Matrices.If A Cnis Hermitian, thenR(A) =R(A) and Ker(A) =Ker(A). Theseequalitiesremaintrueforalargerclassofmatrices, theso-callednormalmatrices. Amatrix A Cnis normal if AA =AA.Show: If A Cnnis normal, thenR(A) =R(A) and Ker(A) =Ker(A).Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 90iiiiiiiiSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 91iiiiiiii5. Least Squares ProblemsHere we solve linear systemsAx = b that do not have a solution. Ifb is not inthe column space ofA, there is nox such thatAx = b. The best we can do is tond a vectory that brings left- and right-hand sides of the linear system as closeas possible; in other words y is chosen to make the distance between Ay and b assmall as possible. That is, we want to minimize the distance Ax b2 over all x,and distance will again be measured in the two norm.Denition 5.1 (Least Squares Problem). LetA Cmnandb Cm. The leastsquares problem consists of nding a vector y Cnso thatminxAx b2=Ay b2.The vector Ay b is called the least squares residual.The name comes about as follows:minxAx b22= minx.,,.least

i|(Ax b)i|2. ,, .squares.5.1 Solutions of Least Squares ProblemsWe express the solutions of least squares problems in terms of the SVD.Let A Cmnhave rank(A) =r and an SVDA =U_

r00 0_V, U = _r mrUrUmr_, V = _r nrVrVnr_,whereU CmmandV Cnnare unitary,andris a diagonal matrix withdiagonal elements 1 r> 0, i.e., ris nonsingular.91Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 92iiiiiiii92 5. Least Squares ProblemsFact 5.2 (All Least Squares Solutions). LetA Cmnandb Cm. The solu-tions of minxAx b2 are of the formy =Vr

1rUr b+Vnrz for any z Cnr.Proof.Let y be a solution of the least squares problem, partitionVy =_VryVnry_=_wz_,and substitute the SVD of A into the residual,Ay b =U_

r00 0_Vy b =U_

rwUr bUmrb_.Two norms are invariant under multiplication by unitary matrices, so thatAy b22=rwUr b22+Umrb22.Since the second summand is constant and independent ofw andz, the residualis minimized if the rst summand is zero, that is, if w =1rUr b. Therefore, thesolution of the least squares problem equalsy =V_wz_=Vrw+Vnrz =Vr

1rUr b +Vnrz.Fact 4.20 implies thatVnrz Ker(A) for any vectorz. HenceVnrz does nothave any effect on the least squares residual, so that z can assume any value.Fact 5.1 shows that if A has rank r< n, then the least squares problem hasinnitely many solutions. The rst term in a least squares solution contains thematrixVr

1rUr =V_

1r00 0_Uwhich is obtained by inverting only the nonsingular parts of an SVD. This matrixis almost an inverse, but not quite.Denition 5.3 (MoorePenrose Inverse).If A Cmnand rank(A) =r 1, letA =U_

r00 0_V be an SVD where ris nonsingular.The nm matrixA=V_

1r00 0_Uis called MoorePenrose inverse of A.If A =0mn, then A=0nm.The MoorePenrose inverse of a full rank matrix can be expressed in termsof the matrix itself.Remark 5.4 (MoorePenrose Inverses of Full Rank Matrices).Let A Cmn.If A is nonsingular, then A=A1.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 93iiiiiiii5.1. Solutions of Least Squares Problems 93If A Cmnand rank(A) =n, then A=(AA)1A.This means AA =In, so that Ais a left inverse of A.If A Cmnand rank(A) =m, then A=A(AA)1.This means AA=Im, so that Ais a right inverse of A.Nowwecanexpresstheleast squaressolutionsintermsoftheMoorePenrose inverse, without reference to the SVD.Corollary 5.5 (All Least Squares Solutions).Let A Cmnand b Cmn.Thesolutions of minxAx b2 are of the form y =Ab +q, where q Ker(A).Proof. This follows from setting q =Vnrz Ker(A) in Fact 4.20.Althoughaleastsquaresproblemcanhaveinnitelymanysolutions, allsolutions have the part Ab in common, and they differ only in the part that belongsto Ker(A). As a result, all least squares solutions have not just residuals of thesame norm, but they have the same residual.Fact 5.6 (Uniqueness of the Least Squares Residual). Let ACmnandb Cm. AllsolutionsyofminxAx b2havethesameresidual b Ay =(I AA)b.Proof. Let y1andy2besolutionstominxAx b2. Corollary5.5impliesy1=Ab +q1 and y2=Ab +q2, where q1, q2 Ker(A).Hence Ay1=AAb =Ay2, and both solutions have the same residual, b Ay1 =b Ay2 =(I AA)b.Besides being unique, the least squares residual has another important prop-erty:It is orthogonal to the column space of the matrix.Fact 5.7 (Residual is Orthogonal to Column Space). Let A Cmn, b Cm,and y a solution of minxAx b2 with residual r =b Ay. Then Ar =0.Proof. Fact 5.6 implies that the unique residual isr = (I AA)b. LetA havean SVDA =U_

r00 0_V,whereUandVare unitary, andris a diagonal matrix with positive diagonalelements.From Denition 5.3 of the MoorePenrose inverse we obtainAA=U_Ir00 0(mr)(mr)_U, I AA=U_0rr00 Imr_U.Hence A(I AA) =0nm and Ar =0.The part of the least squares problemsolution y =Ab+q that is responsibleforlackofuniquenessisthetermq Ker(A). WecanforcetheleastsquaresSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 94iiiiiiii94 5. Least Squares Problemsproblem to have a unique solution if we add the constraint q =0.It turns out thatthe resulting solution Ab has minimal norm among all least squares solutions.Fact 5.8 (Minimal Norm Least Squares Solution). Let A Cmnand b Cm.Among all solutions of minxAx b2 the one with minimal two normis y =Ab.Proof. From the proof of Fact 5.2 follows that any least squares solution has theformy =V_

1rUr bz_.Hencey22=1rUr b22+z221rUr b22=Vr

1rUr b22=Ab22.Thus, any least squares solution y satises y2 Ab2. This means y =Abis the least squares solution with minimal two norm.The most pleasant least squares problems are those where the matrix A hasfull columnrankbecause thenKer(A) ={0} andthe least squares solutionis unique.Fact 5.9 (Full Column Rank Least Squares). Let A Cmnandb Cm. Ifrank(A) =n, then minxAx b2 has the unique solution y =(AA)1Ab.Proof. From Fact 4.22 we know that rank(A) = n implies Ker(A) = {0}. Henceq =0 in Corollary 5.5. The expression for Afollows from Remark 5.4.In particular,whenA is nonsingular,then the MoorePenrose inverse re-duces to the ordinary inverse. This means,if we solve a least squares problemminxAx b2 with a nonsingular matrix A, we obtain the solution y =A1b ofthe linear system Ax =b.Exercises(i) What is the MoorePenrose inverse of a nonzerocolumnvector? of a nonzerorow vector?(ii) Let u Cmnand v Cnwith v =0.Show that uv

2=u2/v2.(iii) Let A Cmn.Show that the following matrices are idempotent:AA, AA, ImAA, InAA.(iv) Let A Cmn.Show:If A =0, then AA

2=AA2=1.(v) Let A Cmn.Show:(ImAA)A =0mn, A(InAA) =0mn.(vi) Let A Cmn.Show: R(A) =R(A) and Ker(A) =Ker(A).Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 95iiiiiiii5.2. Conditioning of Least Squares Problems 95(vii) Let A Cmnhave rank(A) =r.Show: A

2=1/r.(viii) Let A Cmnhave rank(A) =n.Show: (AA)1

2=A

22.(ix) Let A=BC where B Cmnhas rank(B) =n and C Cnnis nonsingular.Show: A=C1B.(x) Let A Cmnwith rank(A) =n and thin QR factorization A=QR, whereQQ=In and R is upper triangular.Show: A=R1Q.(xi) Show:If A has orthonormal columns, then A=A.(xii) Partial Isometry.A matrixA Cmnis called a partial isometry ifA= A. Show: A is apartial isometry if and only if all its singular values are 0 or 1.(xiii) What is the minimal norm solution to minxAx b2 when A =0?(xiv) If y is the minimal normsolution to minxAx b2 and Ab =0, then whatcan you say about y?(xv) Given an approximate solutionz to a linear systemAx = b, this problemshows howto construct a linear system(A+E)x =b for which z is the exactsolution.Let A Cmnand b Cm.Let z Cnwith z =0 and residual r =b Az.Show:If E =rz, then (A+E)z =b.1. What is the minimal normsolution to minxAx b2 when A=uv, whereu and v are column vectors?2. Let ACmn. Show: The singular values of_InA_are equal to 1/_1+2j ,1 j n.3. Let A Cmnhave rank(A) =n.Show: I AA

2=min{1, mn}.4. Let A Cmn. Show: Ais the MoorePenrose inverse of A if and only ifAsatisesMP1: AAA =A, AAA=A,MP2: AAand AA are Hermitian.5. Partitioned MoorePenrose Inverse.Let ACmnhave rank(A) =nandbe partitionedas A=_A1A2_. Show:(a)A=_B1B2_, where B1=(I A2A2)A1, B2=(I A1A1)A2.(b) B1

2=minZA1A2Z2 and B2

2=minZA2A1Z2.(c) Let 1 k n, and let V11 be the leading k k principal submatrix of V.Show:If V11 is nonsingular, then A1

2V111 2/k.5.2 Conditioning of Least Squares ProblemsLeast squares problems are much more sensitive to perturbations than linear sys-tems. A leastsquaresproblemwhosematrixisdecientincolumnrankissoSociety for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 96iiiiiiii96 5. Least Squares Problemssensitivethatwecannotevendeneaconditionnumber. Theexamplebelowillustrates this.Example 5.10 (Rank Decient Least Squares Problems are Ill-Posed). Con-sider the least squares problem minxAx b2 withA =_1 00 0_=A, b =_11_, y =Ab =_10_.The matrix A is rank decient and y is the minimal norm solution.Let us perturbthe matrix so thatA+E =_1 00 _, where 0 0.Society for Industrial and Applied MathematicsCopyright 2009 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed.Buy this book from SIAM at www.ec-securehost.com/SIAM/OT113.htmlbook2009/5/27page 108iiiiiiii108 6. SubspacesFact 6.6 (Column Space). If A Cmnand A =0, then R(A) =R(Ur).Proof. In the reduced SVDA = Ur

rVr , the matricesVrandrhave linearlyindependent rows.Fact 6.3 implies R(A) =R(Ur).Fact 6.7 (Null Space). If ACmnandrank(A) =r .Since S =R(UA) =R(UB), eachof thecolumns ofUAcan be expressed as a linear combination ofUB. Thismeans UA=UBY, where Y is a matrix. Using the fact that UA and UB haveorthonormal columns givesI =UAUA=YUBUBY =YY.Fact 4.14 and Example 4.8 imply =rank(I) =rank(YY) =rank(Y) min{, } =.Thus , which contradicts the assumption> . Therefore, we must have =, so that the dimension of S is unique.The so-called dimension formula below is sometimes called the rst part ofthe fundamental theorem of linear algebra.The formula relates the dimensionsof column and null spaces to the number of rows and columns.Fact 6.10 (Dimension Formula). If A Cmn, thenrank(A) =dim(R(A)) =dim(R(A))andn =rank(A) +dim(Ker(A)), m=rank(A) +dim(Ker(A)).Society for Industrial and Applied MathematicsCopyright 2009