MA 580; Gaussian Elimination - Nc State University · 2018. 7. 23. · Name of method: Gaussian...
Transcript of MA 580; Gaussian Elimination - Nc State University · 2018. 7. 23. · Name of method: Gaussian...
Part IVa: Gaussian Elimination
MA 580; Gaussian Elimination
C. T. KelleyNC State University
Version of September 7, 2016
Master Chapters 1--7 of the Matlab book. NOW!!!
Read chapter 5 of the notes. Start on Chapter 9
of the Matlab book.
NCSU, Fall 2016Part IVa: Gaussian Elimination
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 1 / 78
Part IVa: Gaussian Elimination
Simple Gaussian Elimination Revisited
For the second time we solve
(I) 2x1 + x2 = 1(II) x1 + x2 = 2
This time we will record what we do.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 2 / 78
Part IVa: Gaussian Elimination
Pivot element and multiplier
We plan to multiply (I) by .5 and subtract from (II).Words:
a11 is the pivot element
l21 = .5 = a21/a11 is the multiplier
I will tell you what L is soon.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 3 / 78
Part IVa: Gaussian Elimination
Solving the system
As before: the new (II) is
(II)’x2/2 = 1.5
Now I can backsolve the problem starting with x2 = 3.Record the coefficient of x2 in (II)’ as u22 = .5.Now plug into (I) to get x1 = −1.In matrix form, there’s something profound happening.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 4 / 78
Part IVa: Gaussian Elimination
Encoding the history of Gaussian Elimination
Suppose I put the multipliers in a unit lower triangular matrix L
L =
(1 0
l21 1
)=
(1 0.5 1
)and the coefficients of the backsolve in
U =
(2 10 .5
).
I’ve encoded all the work in the solve in L and U. In fact
LU =
(1 0.5 1
)(2 10 .5
)=
(2 11 1
)= A
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 5 / 78
Part IVa: Gaussian Elimination
The Factor-Solve Paradigm
In real life, one separates the factorization A = LU from the solve.First factor A = LU.Then
Solve Lz = b (forward solve)
Solve Ux = z (back solve)
Put this together and
Ax = LUx = Lz = b.
So you can do several right hand side vectors with a singlefactorization.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 6 / 78
Part IVa: Gaussian Elimination
Triangular Solves
The system for the forward solve looks like
Lz =
1 0 0 . . . 0
l21 1 0 . . . 0l31 l32 1 0 . . ....
......
. . . 0lN1 lN2 lN3 . . . 1
z1z2z3...
zN
=
b1
b2
b3...
bN
So z1 = b1, z2 = b2 − l21z1, ...
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 7 / 78
Part IVa: Gaussian Elimination
Algorithm for Lower Triangular Solve (forward)
Forward means solve for zi in orderSolve Lz = b with unit triangular L
for i = 1 : N dozi = bi
for j = 1 : i − 1 dozi = zi − lijzj
end forend for
Bottom line: zi = bi −i−1∑j=1
lijzj , 1 ≤ i ≤ N.
What if lii 6= 1?
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 8 / 78
Part IVa: Gaussian Elimination
Algorithm for Upper Triangular Solve (backward)
Backward means solve for zi in reverse order (zN first)Solve Ux = z
for i = N : 1 doxi = zifor j = i + 1 : N do
xi = xi − uijxjend forxi = xi/uii
end for
Bottom line: xi =
zi −N∑
j=i+1
uijxj
/uii , N ≥ i ≥ 1
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 9 / 78
Part IVa: Gaussian Elimination
Evaluating Cost
You can use several equivalent metrics for cost
floating point operations
floating point multiplications
“flops”an add + a multiply + some address computation
In MA 580, with one exception, adds and multiplies are paired andthe three metrics give equal relative results.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 10 / 78
Part IVa: Gaussian Elimination
Scalar products and matrix multiplies
xTy =N∑i=1
xiyi
N multiplies and N − 1 adds.Cost: N + O(1) leading order is the same for the adds/multipliesMatrix-vector product:
(Ax)i =N∑i=1
aijxj for j = 1 . . .N
N2 multiplies and N2 − N adds (N scalar products)Cost: N2 + O(N)
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 11 / 78
Part IVa: Gaussian Elimination
A trick
Write the loops as nested sums and add 1 for each multiply.
Replace the sums by integrals.
Evaluate the integrals
You’ll be right to leading order.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 12 / 78
Part IVa: Gaussian Elimination
Triangular Solve
We’ll do lower, upper is the same.Algorithm:
for i=1:N dozi = bi −
∑i−1j=1 lij ∗ zj
end for
Sum:N∑i=1
i−1∑j=1
1
Integral (check it out)∫ N
1
∫ i−1
1dj di = N2/2 + O(N)
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 13 / 78
Part IVa: Gaussian Elimination
Simple Gaussian Elimination
Here are the steps
Make a copy of A.
Do Gaussian elimination as if one were solving an equation.
Recover L and U from our (mangled) copy of A.
In the real world, we would overwrite A with the data for thefactors, saving storage.Have I told you about the computer scientist who was low onmemory?The guy tells me he doesn’t have enough bytes.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 14 / 78
Part IVa: Gaussian Elimination
Not enough bytes?
So I bit him.H. Youngman
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 15 / 78
Part IVa: Gaussian Elimination
Overwriting A with L and U
lu simple A← A
for j = 1 : N dofor i = j + 1 : N do
aij = aij/ajj {Compute the multipliers. Store them in thestrict lower triangle of A.}for k = j + 1 : N do
aik = aik − aijajk {Do the elimination.}end for
end forend for
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 16 / 78
Part IVa: Gaussian Elimination
Cost of LU factorization
N∑j=1
N∑i=j+1
N∑k=j+1
1 ≈∫ N
1
∫ N
j+1
∫ N
j+1dk di dj = N3/3 + O(N2).
So, cost of LU is N3/3 + O(N2).
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 17 / 78
Part IVa: Gaussian Elimination
Recovering L and U
(L,U)← recover(A)
for j = 1 : N doljj = 1for i = 1 : j do
uij = aijend forfor i = j + 1 : N do
lij = aijend for
end for
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 18 / 78
Part IVa: Gaussian Elimination
Factor-Solve Paradigm: II
The factorization costs O(N3) work
The upper and lower triangular solves cost O(N2) work
So . . .
the standard procedure is to separate them.This makes is much more efficient to handle multiple righthand sides,as we will do in the nonlinear equations part.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 19 / 78
Part IVa: Gaussian Elimination
Mission Accomplished?
Not exactly. Does
A =
(0 11 0
)have an LU factorization?No, but solving
Ax =
(x2x1
)= b =
(b1
b2
)is pretty easy. We’ve clearly missed something.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 20 / 78
Part IVa: Gaussian Elimination
Partial pivoting aka column pivoting
The easy fix it to interchange the equations (rows) in the problem.Then you get A′ = b′.
A′ =
(1 00 1
)and b′ = (b2, b1)T .Same answer, different problem.Proposed algorithm: When you get a zero pivot, find somethingbelow it that is not zero, and swap those rows. Then make sureyou don’t forget what you did.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 21 / 78
Part IVa: Gaussian Elimination
Are we there yet?
How about
A =
(.1 101 0
)and b = (20,−1)T .Exact solution: (−1, 2.01)T
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 22 / 78
Part IVa: Gaussian Elimination
Recall the Toy Floating Point Number System
Here’s a radix 10 system.
Two digit mantissa
Exponent range: -2:2
Largest float = .99 ∗ 102 = 99
eps(0) = .10 ∗ 10−2
eps(1) = 1.1− 1 = .1
Round to nearest.
fl(x∗) = fl((−1, 2.01)T ) = (−1, 2)T .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 23 / 78
Part IVa: Gaussian Elimination
How about some elimination?
Multiply first equation by 10 and subtract from second:
.1x+ y = 20−100y = fl(−1− 200) = overflow!! = INF .
Better fix: find largest element in absolute value and swap rows.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 24 / 78
Part IVa: Gaussian Elimination
Swap rows and it’s better.
x = −1.1x +y = 20
Multiply first equation by .1 and subtract from second to get
x = −110y = 20
which does the right thing.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 25 / 78
Part IVa: Gaussian Elimination
What did we do right?
We normalized correctly by choosing the pivot element wisely.Name of method: Gaussian Elimination with Partial Pivoting(GEPP)aka: Gaussian Elimination with Column PivotingThis is what the MATLAB backslash command does.
x = A\b;
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 26 / 78
Part IVa: Gaussian Elimination
GEPP
lu gepp A← A
for j = 1 : N doFind k so that akj = maxi≥j |aij |Swap rows j and k ; record the swapfor i = j + 1 : N do
aij = aij/ajj {Compute the multipliers. Store them in thestrict lower triangle of A.}for k = j + 1 : N do
aik = aik − aijajk {Do the elimination.}end for
end forend for
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 27 / 78
Part IVa: Gaussian Elimination
Matlab for the elimination
One way (clarity)
for k=j+1:N
a(i,k)=a(i,k)-a(i,j)*a(j,k);
end
Other way: vectorize (faster)
a(i,j+1:N) = a(i,j+1:N) - a(i,j)*a(j,j+1:N);
Even better: make i the inner loop index and vectorize that.
Why? Matlab (like fortran) stores arrays by columns.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 28 / 78
Part IVa: Gaussian Elimination
Does this really make any difference?
Let’s find out:We will compute the multipliers first
for i=j+1:n
a(i,j)=a(i,j)/a(j,j);
end
and then try four different things and compare to Matlab’s lu.Matrix is a = rand(1000,1000).
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 29 / 78
Part IVa: Gaussian Elimination
Non-vectorized vs vectorized. i loop outside
Version 1
for i=j+1:n
for k=j+1:n
a(i,k)=a(i,k)-a(i,j)*a(j,k);
end
end
Version 2
for i=j+1:n
a(i,j+1:n) = a(i,j+1:n) - a(i,j)*a(j,j+1:n);
end
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 30 / 78
Part IVa: Gaussian Elimination
k loop outside. Vectorized vs totally vectorized
Version 3
for k=j+1:n
a(j+1:n,k)=a(j+1:n,k)-a(j+1:n,j)*a(j,k);
end
and, now for something completely differentVersion 4
a(j+1:n,j+1:n)=a(j+1:n,j+1:n)-a(j+1:n,j)*a(j,j+1:n)
;
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 31 / 78
Part IVa: Gaussian Elimination
Bottom line
Timings in seconds via Matlab tic and toc.
Version 1 Version 2 Version 3 Version 4 Matlab lu
25 10 5 3 1.58e-02
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 32 / 78
Part IVa: Gaussian Elimination
Back to business: with GEPP you’ve stabilized L
By construction
|lij | ≤ 1
So ‖L‖∞, the maximum row sum ≤ Nwhich is more than small enough.
But you have NOT stabilized U.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 33 / 78
Part IVa: Gaussian Elimination
Incremental Cost of GEPP
Swapping rows is index manipulation and copying. Neglect.
Record swaps by permuting an integer vector. Neglect.
A floating point comparison is a subtract followed by lookingat the sign bit. Cost = one add.
Total number of adds:
N∑j=1
(N − j) =
∫ N
1(N − j) dj + O(N) =
N2
2+ O(N).
Negligible compared to N3/3 cost (in paired mult/adds) for theelimination.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 34 / 78
Part IVa: Gaussian Elimination
GEPP as Matrix Factorization
If record the swap means build a matrix P so that
P = I at the start
You swap columns of P as you swap rows of A.
ThenA = PLU
P is a permutation matrix.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 35 / 78
Part IVa: Gaussian Elimination
Permutation Matrices
Let Pij be the identity with rows i and j swapped.Pij also swaps columns i and j .
The Pijs do not commute.
A permutation matrix is a product of some of the Pijs
A permutation matrix P is orthogonal. PT = P−1.
A sequence of column swaps of I is the transpose of asequence of row swaps.
PijA swaps rows; APij swaps columns.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 36 / 78
Part IVa: Gaussian Elimination
Some MATLAB: I
When you call Matlab’s lu code, the permutation is wrapped in l .
>> A=[1 2 3; 4 5 6; 7 8 9];
>> [l,u]=lu(A);
>> l
l =
1.4286e-01 1.0000e+00 0
5.7143e-01 5.0000e-01 1.0000e+00
1.0000e+00 0 0
Note the use for short e format. Put format short e in yourMatlab startup file.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 37 / 78
Part IVa: Gaussian Elimination
Some MATLAB: II
I’ve put an explicit GEPP file on the moodle pageplu.mfor you to play with.
>> A=[1 2 3; 4 5 6; 7 8 9];
>> [l, u, p]=plu(A);
>> [p p*l]
ans =
0 1 0 1.4286e-01 1.0000e+00 0
0 0 1 5.7143e-01 5.0000e-01 1.0000e+00
1 0 0 1.0000e+00 0 0
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 38 / 78
Part IVa: Gaussian Elimination
Errors in PLU factorization
Recall: solve is Lz = PTb followed by Ux = z. Define
E = PLU− A, rL = Lz− PTb, rU = Ux− z
Goal: estimate‖x∗ − x‖∞‖x‖∞
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 39 / 78
Part IVa: Gaussian Elimination
Building the bound
Define some (hopefully small) errors
εA =‖E‖∞‖A‖∞
, εL =‖rL‖∞
‖L‖∞‖z‖∞, εR =
‖rU‖∞‖U‖∞‖x‖∞
,
and the growth factor
g = ‖U‖∞/‖A‖∞
which measures element growth in the elimination.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 40 / 78
Part IVa: Gaussian Elimination
Error Bound
‖x∗ − x‖∞‖x‖∞
≤ κ∞(A) [εA + Ng(εU(1 + εL) + εL)]
The bad news here could be g and/or κ. Errors in triangular solvesare benign.Proof in Chapter 3 of pink book.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 41 / 78
Part IVa: Gaussian Elimination
Instability of GEPP and the evil matrix
Here’s the matrix that breaks GEPP
A =
1 0 . . . 0 0 1−1 1 0 . . . 0 1
−1 −1 1 0 0...
.... . .
. . .. . . 0
...−1 . . . . . . −1 1 1−1 . . . . . . . . . −1 1
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 42 / 78
Part IVa: Gaussian Elimination
Coefficients for the evil matrix
aij =
1 if i = j or j = N−1 if i > j
0 otherwise.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 43 / 78
Part IVa: Gaussian Elimination
Example: N=5
A =
1 0 0 0 1−1 1 0 0 1−1 −1 1 0 1−1 −1 −1 1 1−1 −1 −1 −1 1
No pivoting necessary. Just add row 1 to the other rows and . . .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 44 / 78
Part IVa: Gaussian Elimination
First step of GEPP
A←
1 0 0 0 10 1 0 0 20 −1 1 0 20 −1 −1 1 20 −1 −1 −1 2
No pivoting necessary. Just add row 2 to the following rows and. . .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 45 / 78
Part IVa: Gaussian Elimination
Second step of GEPP
A←
1 0 0 0 10 1 0 0 20 0 1 0 40 0 −1 1 40 0 −1 −1 4
No pivoting necessary. Just add row 3 to the following rows and. . .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 46 / 78
Part IVa: Gaussian Elimination
Third step of GEPP
This is not looking too good.
A←
1 0 0 0 10 1 0 0 20 0 1 0 40 0 0 1 80 0 0 −1 8
No pivoting necessary. Just add row 4 to the following rows and. . .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 47 / 78
Part IVa: Gaussian Elimination
Fourth step of GEPP
What a stinker!
A←
1 0 0 0 10 1 0 0 20 0 1 0 40 0 0 1 80 0 0 0 16
Looks like ‖U‖∞ = 2N−1. For this matrix ‖A‖∞ = N, so thegrowth factor gPP for GEPP is kinda big . . .
gPP = 2N−1/N.
GEPP is unstable for this problem, as you can (and should) checkwith Matlab’s lu command.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 48 / 78
Part IVa: Gaussian Elimination
If GEPP is unstable, why do we use it?
BECAUSE IT WORKS!
The pathology of the example is never seen in practice
Any fix, in the opinion of the whole world, costs too much,including the fixes we will propose in this course.
This is our first example of an algorithm that can be unstable,but rarely or never is in practice.
How can you fail to love this stuff?
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 49 / 78
Part IVa: Gaussian Elimination
Matlab for the evil matrix
function a = evil(n)
% EVIL Create the n x n evil matrix to destabilize
GEPP
% a = evil(n)
%
a=eye(n,n);
for i=1:n
a(i,i)=1;
a(i,n)=1;
for j=1:i-1
a(i,j)=-1;
end
end
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 50 / 78
Part IVa: Gaussian Elimination
Conditioning and growth factor
Use Matlab’s cond command.
>> a=evil(5);
>> kappa=cond(a,inf);
>> [l,u]=lu(a);
>> [kappa, norm(u,inf)]
ans =
5 16
>>
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 51 / 78
Part IVa: Gaussian Elimination
Look at κ and gPP as N increases.
function stats = evil_study
% EVIL_STUDY Look at conditioning and growth factor
% records kappa and growth factor in stats array
%
dimensions =[10 20 50 100 500 1000];
stats=zeros (6,3);
for id=1:6
stats(1,id)=dimensions(id);
a=evil(dimensions(id));
stats(id ,2)=cond(a,2); % record kappa_2
stats(id ,3)=cond(a,inf); % record kappa_inf
[l,u]=lu(a);
stats(id ,4)=norm(u,inf)/dimensions(id);
end
stats
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 52 / 78
Part IVa: Gaussian Elimination
Something’s fishy and poorly presented here
>> evil_study;
stats =
1.0000e+01 4.3850e+00 1.0000e+01 5.1200e+01
2.0000e+01 8.8343e+00 2.0000e+01 2.6214e+04
5.0000e+01 2.2306e+01 5.0000e+01 1.1259e+13
1.0000e+02 4.4802e+01 1.0000e+02 6.3383e+27
5.0000e+02 2.2486e+02 2.2397e+105 3.2734e+147
1.0000e+03 4.4993e+02 3.3018e+271 5.3575e+297
Turn in a table like this and get zero.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 53 / 78
Part IVa: Gaussian Elimination
Presentation of results
N κ2 κ∞ gPP10 4.38e+00 1.00e+01 5.12e+0120 8.83e+00 2.00e+01 2.62e+0450 2.23e+01 5.00e+01 1.13e+13100 4.48e+01 1.00e+02 6.34e+27500 2.25e+02 2.24e+105 3.27e+1471000 4.50e+02 3.30e+271 5.36e+297
Can this possibly be correct?
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 54 / 78
Part IVa: Gaussian Elimination
Compare norms
Note that‖x2‖/
√N ≤ ‖x‖∞ ≤ ‖x‖2.
Can you prove this?So
‖Ax‖∞ ≤ ‖Ax‖2 ≤ ‖A‖2‖x‖2 ≤√
N‖A‖2‖‖x‖∞.
Which implies that
‖A‖∞ ≤√
N‖A‖2 and so κ∞(A) ≤ Nκ2(A).
The κ∞ column in the table is wrong!
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 55 / 78
Part IVa: Gaussian Elimination
What happened?
The `∞ condition estimator in Matlab uses the LUfactorization.
So the instability affects the estimate and it’s wrong.
Bottom line: the evil matrix is not badly conditioned.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 56 / 78
Part IVa: Gaussian Elimination
Complete Pivoting
We swapped rows to stabilize computation of L.
How about swapping columns for U?
When done correctly, this is complete pivoting.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 57 / 78
Part IVa: Gaussian Elimination
Complete Pivoting Algorithm
Pick the pivot element for column j by making swapping
Row j and row k
Column j and column l
where|akl | = max
m,n≥j|amn|
Contrast with partial pivoting, where you only search over the rowindex.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 58 / 78
Part IVa: Gaussian Elimination
lu gecp
A← A
for j = 1 : N doFind k , l so that akl = maxm,n≥j |amn|Swap rows j and k , columns j and l ; record the swapsfor i = j + 1 : N do
aij = aij/ajj {Compute the multipliers. Store them in thestrict lower triangle of A.}for k = j + 1 : N do
aik = aik − aijajk {Do the elimination.}end for
end forend for
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 59 / 78
Part IVa: Gaussian Elimination
Incremental Cost
At stage j in the outer loop you do (N − j)2 compares,
which is equivalent to (N − j)2 adds.
So you doing∫ N
1(N − j)2 dj = N3/3 + O(N2) extra adds
This is the example where adds and multiplies are not pairedto leading order.
If adds and multiplies cost the same, net increase of 50%.Not really worth it.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 60 / 78
Part IVa: Gaussian Elimination
GECP as a matrix factorization
Here we get A = PLLUPR where PL and PR are permutationmatrices.The solve goes like
Multiply Ax = b by PTL to get
LUPRx = PLb ≡ b′
Solve Lz = b′
Solve Uw = z
Set x = PTR w
So . . .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 61 / 78
Part IVa: Gaussian Elimination
Just to make sure
Ax = PLLUPRx = PLLUPRPTR w
= PLLUw = PLLz
= PLb′ = PLPTL b = b
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 62 / 78
Part IVa: Gaussian Elimination
Growth factor for GECP
Here’s a fact.gCP ≈ N .5+ln(N/4)
which is much better than 2N−1/N.The conjecture is that gCP = O(N).Prove that and get PhD and cool job right now!
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 63 / 78
Part IVa: Gaussian Elimination
Uniqueness of LU factorization: I
Put all pivots as part of A. Replace A by PTL APT
R .For unit lower triangular L and upper triangular U, the equation
LU = A
is a system of N2 equations in N2 unknowns that is easy to solve.To start, since l11 = 1, the equation for the first row is
u1j = a1j , 1 ≤ j ≤ N.
so the first row of U is the first row of A.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 64 / 78
Part IVa: Gaussian Elimination
Uniqueness of LU factorization: II
For the second row, note that l22 = 1 and l2k = 0 if k > 2.
a2j =∑
k=1:N l2kukj =∑
k=1:2 l2kukj
= l21u1j + l11u2j
So, for 1 ≤ j ≤ N,u2j = a2j − l21u1j
and so on.I may well ask you to prove things like this in detail on an exam.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 65 / 78
Part IVa: Gaussian Elimination
Integral Equation Example
We will use the trapezoid rule to solve
u(x)−∫ 1
0sin(x − y)u(y) dy = f (x).
We will test the quality of the results with
the method of manufactured solutions and
a grid refinement study.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 66 / 78
Part IVa: Gaussian Elimination
The method of manufactured solutions
Pick x∗ and set b = Ax∗.
Solve Ax = b and obtain x̃.
Compare x̃ to x∗.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 67 / 78
Part IVa: Gaussian Elimination
For example . . .
Here’s a function that examines a 10× 10 matrix A.
function ediff = testme(A)
% TESTME sees if the matlab backslash can
solve an easy problem.
%
xstar = ones (10,1);
bstar=A*xstar;
xtilde=A\bstar;
ediff = norm(xtilde - xstar ,1);
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 68 / 78
Part IVa: Gaussian Elimination
Composite Trapezoid Rule
∫ 1
0u(x) dx ≈ Ih(u) ≡
N∑i=1
wiu(xi )
where
h = 1/(N − 1); xi = (i − 1) ∗ h; wi =
{h/2 i = 1 or i = Nh 2 ≤ i ≤ N − 1
Theorem: If u is sufficiently smooth the the composite trapezoidrule is second order accurate.
Ih(u)−∫ 1
0u(x) dx = O(h2).
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 69 / 78
Part IVa: Gaussian Elimination
Approximate the integral equation
ui −N∑j=1
∫ 1
0sin(xi − xj)wjuj = f (xi ).
Theorem: The integral equation has a unique solution u∗ for everycontinuous f . The approximate problem has a unique solution uh
for each N. u∗ is as differentiable as f . And if f is sufficientlysmooth then
max1≤i≤N
|ui − u∗(xi )| = O(h2)
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 70 / 78
Part IVa: Gaussian Elimination
Building the Approximation
For a given N and h define
ahij = δij − sin(xi − xj)wj and f hi = f (xi ).
The approximate integral equation is Ahuh = fh.Let uh∗
i = u(xi ), where u is the solution of the integral equation.The error estimate is
Eh ≡ ‖uh − uh∗‖∞ = O(h2).
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 71 / 78
Part IVa: Gaussian Elimination
Grid refinement study
If you knew u∗, you could compute u∗h and therefore Eh.You’d expect
E2h/Eh ≈ 4. Why?
Let’s check that.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 72 / 78
Part IVa: Gaussian Elimination
Method of Manufactured Solutions
Set u∗(x) ≡ 1. Then
f (x) = 1−∫ 1
0sin(x − y) dy = 1− cos(x − 1) + cos(x)
Now for the computation . . .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 73 / 78
Part IVa: Gaussian Elimination
Grids, A, F
Cycle through
h = 1/(10× 2p), 1 ≤ p ≤ 5So N = 21, 41, 81, . . . 321.
Loop over p and print out E (2h)/E (h)
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 74 / 78
Part IVa: Gaussian Elimination
Some matlab
We’ll get the weights and nodes right.
for p=1:5
N=(10*2^p) + 1; h=1/(N-1);
w=h*ones(N,1); w(1)=w(1) *.5; w(N) = w(N)
*.5;
%
% Make sure the spatial grid points are in a
column vector.
%
x=(0:N-1)*h; x=x’;
f=ones(N,1) -cos(x-1)+cos(x);
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 75 / 78
Part IVa: Gaussian Elimination
Building the matrix and solving the problem
ustar=ones(N,1);
A=eye(N,N);
for j=1:N
for i=1:N
A(i,j) = A(i,j) - sin(h*(i-j))*w(j);
end
end
utest=A\f;
EH(p,2)=norm(utest -ustar ,inf);
EH(p,1)=N;
end % end of p loop
EH(2:5 ,3)=EH(1:4 ,2)./EH(2:5 ,2);
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 76 / 78
Part IVa: Gaussian Elimination
Tastefully presented output
N E(h) E(2h)/E(h)
21 1.02e-0441 2.56e-05 4.00098e+0081 6.39e-06 4.00025e+00161 1.60e-06 4.00006e+00321 3.99e-07 4.00002e+00
Code is integral example.m on Moodle page.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 77 / 78
Part IVa: Gaussian Elimination
What have we done?
You just learned to solve any 2nd kind Fredholm intergral equation
u(x)−∫ 1
0k(x , y)u(y) dy = f (x).
If there’s a unique solution u∗ for any f then all you do is this.
Build Ah and fh.
Solve Ahuh = fh with Gaussian elimination.
And thenlimh→0
max1≤i≤N
|ui − u∗(xi )| = 0.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IVa: Gaussian Elimination MA 580, Fall 2016 78 / 78