MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.
-
Upload
amanda-wheeler -
Category
Documents
-
view
221 -
download
3
Transcript of MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.
![Page 1: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/1.jpg)
MATH 685/ CSI 700/ OR 682 Lecture Notes
Lecture 3.
Solving linear systems
![Page 2: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/2.jpg)
Solving linear systems To solve linear system, transform it into one whose
solution is same but easier to compute What type of transformation of linear system leaves
solution unchanged? We can premultiply (from left) both sides of linear
system Ax = b by any nonsingular matrix M without affecting solution Solution to MAx =Mb is given by
![Page 3: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/3.jpg)
Permutation matrices
![Page 4: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/4.jpg)
Scaling
![Page 5: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/5.jpg)
Triangular form What type of linear system is easy to solve?
If one equation in system involves only one component of solution (i.e., only one entry in that row of matrix is nonzero), then
that component can be computed by division
If another equation in system involves only one additional solution component, then by substituting one known component into it, we can solve for other component
If this pattern continues, with only one new solution component per equation, then all components of solution can be computed in succession.
System with this property is called triangular
![Page 6: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/6.jpg)
Triangular form
![Page 7: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/7.jpg)
Gaussian elimination
![Page 8: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/8.jpg)
Gaussian elimination
![Page 9: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/9.jpg)
Example
![Page 10: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/10.jpg)
Elimination
![Page 11: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/11.jpg)
Elimination matrices
![Page 12: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/12.jpg)
Elimination matrices
![Page 13: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/13.jpg)
Example
![Page 14: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/14.jpg)
Example
![Page 15: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/15.jpg)
Gaussian elimination
![Page 16: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/16.jpg)
Gaussian elimination
![Page 17: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/17.jpg)
LU decomposition
![Page 18: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/18.jpg)
LU decomposition
![Page 19: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/19.jpg)
Example
![Page 20: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/20.jpg)
Example
![Page 21: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/21.jpg)
Example
![Page 22: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/22.jpg)
Example
![Page 23: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/23.jpg)
Row interchanges Gaussian elimination breaks down if leading diagonal entry of
remaining unreduced matrix is zero at any stage
Easy fix: if diagonal entry in column k is zero, then interchange row k with some subsequent row having nonzero entry in column k and then proceed as usual
If there is no nonzero on or below diagonal in column k, then there is nothing to do at this stage, so skip to next column
Zero on diagonal causes resulting upper triangular matrix U to be singular, but LU factorization can still be completed
Subsequent back-substitution will fail, however, as it should for singular matrix
![Page 24: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/24.jpg)
Partial pivoting In principle, any nonzero value will do as pivot, but in practice
pivot should be chosen to minimize error propagation
To avoid amplifying previous rounding errors when multiplying remaining portion of matrix by elementary elimination matrix, multipliers should not exceed 1 in magnitude
This can be accomplished by choosing entry of largest magnitude on or below diagonal as pivot at each stage
Such partial pivoting is essential in practice for numerically stable implementation of Gaussian elimination for general linear systems
![Page 25: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/25.jpg)
LU with pivoting
![Page 26: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/26.jpg)
Complete pivoting
![Page 27: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/27.jpg)
Example
![Page 28: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/28.jpg)
Small pivots
![Page 29: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/29.jpg)
Small pivots
![Page 30: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/30.jpg)
Pivoting not required sometimes
![Page 31: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/31.jpg)
Residual
![Page 32: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/32.jpg)
Residual
![Page 33: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/33.jpg)
Example
![Page 34: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/34.jpg)
Example
![Page 35: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/35.jpg)
Implementing Gaussian elimination
![Page 36: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/36.jpg)
Complexity
![Page 37: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/37.jpg)
Inversion vs. Factorization
![Page 38: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/38.jpg)
Modified systems If right-hand side of linear system changes but matrix
does not, then LU factorization need not be repeated to solve new system
Only forward- and back-substitution need be repeated for new right-hand side
This is substantial savings in work, since additionaltriangular solutions cost only O(n2) work, in contrast toO(n3) cost of factorization
![Page 39: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/39.jpg)
Rank-one update
![Page 40: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/40.jpg)
Rank-one update
![Page 41: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/41.jpg)
Example
![Page 42: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/42.jpg)
Example
![Page 43: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/43.jpg)
Scaling In principle, solution to linear system is unaffected by diagonal
scaling of matrix and right-hand-side vector
In practice, scaling affects both conditioning of matrix and selection of pivots in Gaussian elimination, which in turn affect numerical accuracy in finite-precision arithmetic
It is usually best if all entries (or uncertainties in entries) of matrix have about same size
Sometimes it may be obvious how to accomplish this by choice of measurement units for variables, but there is no foolproof method for doing so in general
Scaling can introduce rounding errors if not done carefully
![Page 44: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/44.jpg)
Scaling
![Page 45: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/45.jpg)
Iterative refinement
![Page 46: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/46.jpg)
Iterative refinement Iterative refinement requires double storage, since both original
matrix and its LU factorization are required
Due to cancellation, residual usually must be computed with higher precision for iterative refinement to produce meaningful improvement
For these reasons, iterative improvement is often impractical to use routinely, but it can still be useful in some circumstances
For example, iterative refinement can sometimes stabilize otherwise unstable algorithm
![Page 47: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/47.jpg)
SPD matrices
![Page 48: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/48.jpg)
Cholesky decomposition Features of Cholesky algorithm for symmetric positive definite
matrices
All n square roots are of positive numbers, so algorithm is well defined
No pivoting is required to maintain numerical stability Only lower triangle of A is accessed, and hence upper triangular
portion need not be stored Only n3/6 multiplications and similar number of additions are
required
Thus, Cholesky factorization requires only about half work and half storage compared with LU factorization of general matrix by Gaussian elimination, and also avoids need for pivoting
![Page 49: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/49.jpg)
Banded matrices Gaussian elimination for band matrices differs little from general
case — only ranges of loops change Typically matrix is stored in array by diagonals to avoid storing
zero entries If pivoting is required for numerical stability, bandwidth can grow
(but no more than double) General purpose solver for arbitrary bandwidth is similar to code
for Gaussian elimination for general matrices For fixed small bandwidth, band solver can be extremely simple,
especially if pivoting is not required for stability
![Page 50: MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 3. Solving linear systems.](https://reader034.fdocuments.us/reader034/viewer/2022051415/56649d015503460f949d3d61/html5/thumbnails/50.jpg)
Iterative vs. direct methods Gaussian elimination is direct method for solving linear
system, producing exact solution in finite number of steps (in exact arithmetic)
Iterative methods begin with initial guess for solution and successively improve it until desired accuracy attained
In theory, it might take infinite number of iterations to converge to exact solution, but in practice iterations are terminated when residual is as small as desired
For some types of problems, iterative methods have significant advantages over direct methods