LECTURE 5:Systems of linear equations

October 15, 2012

1 Naive Gaussian elimination

Consider the following system of n linear equations with n unknowns:

a11x1 + a12x2 + a13x3 + . . . + a1nxn = b1a21x1 + a22x2 + a23x3 + . . . + a2nxn = b2a31x1 + a32x2 + a33x3 + . . . + a3nxn = b3

......

......

...ai1x1 + ai2x2 + ai3x3 + . . . + ainxn = bi

......

......

...an1x1 + an2x2 + an3x3 + . . . + annxn = bn

In compact form, this system can be written as

n

∑j=1

ai jx j = bi (1≤ i≤ n)

In these equations, ai j and bi are prescribed real numbers (data), and the unknowns x j areto be determined.

The objective in this chapter is to find an efficient way for obtaining the numerical solutionof such a set of linear equations. We begin by discussing the simplest form of a procedureknown as Gaussian elimination. This form is not suitable for automatic computation, butserves as a starting point for developing a more robust algorithm.

1

ExampleWe demonstrate the simplest form of the Gaussian elimination procedure using a numer-ical example. In this form, the procedure is not suitable for numerical computation, but itcan be modified as we see later on.

Consider the following four equations with four unknowns:6x1 − 2x2 + 2x3 + 4x4 = 16

12x1 − 8x2 + 6x3 + 10x4 = 263x1 − 13x2 + 9x3 + 3x4 = −19−6x1 + 4x2 + x3 − 18x4 = −34

In the first round of the procedure, we eliminate x1 from the last three equations. This isdone by subtracting multiples of the first equation from the other ones. The result is

6x1 − 2x2 + 2x3 + 4x4 = 16− 4x2 + 2x3 + 2x4 = −6− 12x2 + 8x3 + x4 = −27

2x2 + 3x3 − 14x4 = −18

Note that the first equation was not changed in the elimination procedure. It is called thepivot equation.

The second step consists of eliminating x2 from the last two equations. We ignore the firstequation and use the second one as the pivot equation. We get

6x1 − 2x2 + 2x3 + 4x4 = 16− 4x2 + 2x3 + 2x4 = −6

+ 2x3 − 5x4 = −94x3 − 13x4 = −21

Finally we eliminate x3 from the last equation and obtain:6x1 − 2x2 + 2x3 + 4x4 = 16

− 4x2 + 2x3 + 2x4 = −6+ 2x3 + 5x4 = −9

− 3x4 = −3

This system is said to be in the upper triangular form. It is equivalent to the system westarted with. This completes the first phase of the Gaussian elimination, called forwardelimination.

The second phase is called back substitution. We solve the system in the upper triangularform, starting from the bottom.

From the last equation we obtain

x4 =−3−3

= 1

Substituting this to the second last equation gives

2x3−5 = 9

Eventually we obtain the solution

x1 = 3 x2 = 1 x3 =−2 x4 = 1

2

AlgorithmWe now write the system of n linear equations in matrix form:

a11 a12 a13 . . . a1na21 a22 a23 . . . a2na31 a32 a33 . . . a3n

......

......

ai1 ai2 ai3 . . . ain...

......

...an1 an2 an3 . . . ann

x1x2x3...xi...

xn

=

b1b2b3...

bi...

bn

or

Ax = b

In the naive Gaussian elimination algorithm, the original data are overwritten with newcomputed values. In the forward elimination phase, there are n−1 principal steps.

The first step consists of subtracting appropriate multiples of the first equation from theothers. The first equation is called the first pivot equation and a11 is called the first pivotelement.

For each of the other equations (2≤ i≤ n), we computeai j ← ai j−

(ai1a11

)a1 j 1≤ j ≤ n

bi ← bi−(

ai1a11

)b1

The quantities ai1/a11 are the multipliers. The new coefficient for x1 in all of the equations2≤ i≤ n will be zero because ai1− (ai1/a11)a11 = 0.

After the first step, the system is in the form

a11 a12 a13 . . . a1n0 a22 a23 . . . a2n0 a32 a33 . . . a3n...

......

...0 ai2 ai3 . . . ain...

......

...0 an2 an3 . . . ann

x1x2x3...xi...

xn

=

b1b2b3...

bi...

bn

From now on, we ignore the first row (and effectively also the first column). We take thesecond row as the pivot equation and compute for the remaining equations (3≤ i≤ n)

ai j ← ai j−(

ai2a22

)a2 j 2≤ j ≤ n

bi ← bi−(

ai2a22

)b2

This step eliminates x2 from the equations 3≤ i≤ n.

3

In the kth step, we use the kth row as the pivot equation and for the remaining equationsbelow the kth row (k+1≤ i≤ n), we compute

ai j ← ai j−(

aikakk

)ak j k ≤ j ≤ n

bi ← bi−(

aikakk

)bk

Note that we must assume that all the divisors in this algorithm are nonzero!!

In the beginning of the back substitution phase, the system is in the upper triangular form.

Back substitution starts by solving the nth equation for xn:

xn =bn

ann

Then using the (n−1)th equation, xn−1 is solved:

xn−1 =1

an−1,n−1(bn−1−an−1,nxn)

The formula for the ith equation is

xi =1aii

(bi−

n

∑j=i+1

ai jx j

)(i = n−1,n−2, . . . ,1)

4

CodeThe following C program performs the Naive Gaussian elimination:

#define N 4

int main(void){

int i, j, k, n;double sum, xmult;double a[N][N], b[N], x[N];FILE *outfile;

/* Read input */GetEquations(a,b);

/* Forward elimination */for(k=0; k<N-1; k++) {

for(i=k+1; i<N; i++) {xmult = a[i][k]/a[k][k];a[i][k] = xmult;for(j=k+1; j<N; j++) {

a[i][j] = a[i][j] - xmult*a[k][j];}b[i] = b[i] - xmult*b[k];

}}

/* Back substitution */x[N-1] = b[N-1]/a[N-1][N-1];for(i=N-2; i>=0; i--) {

sum = b[i];for(j=i+1; j<N; j++) {

sum = sum - a[i][j]*x[j];}x[i] = sum/a[i][i];

}}

5

If we apply the code to the example case discussed before, we get the following output(only recalculated values):

aij bi-4.0000 2.0000 2.0000 -6.0000

-12.0000 8.0000 1.0000 -27.00002.0000 3.0000 -14.0000 -18.0000

2.0000 -5.0000 -9.00004.0000 -13.0000 -21.0000

-3.0000 -3.0000

--------------------------------------------

x[1] = 3.000000x[2] = 1.000000x[3] = -2.000000x[4] = 1.000000

The result is the same as was previously obtained by hand.

This simple algorithm does not work in many cases. The crucial computation is thereplacement

ai j← ai j−(

aik

akk

)ak j

A round-off error in ak j is multiplied by the factor (aik/akk). This factor is large if thepivot element |akk| is small compared to |aik|.

The conclusion is that small pivot elements lead to worse round-off errors.

6

1.1 Condition number

The condition number of a linear system Ax = b is defined as

κ(A) = ||A|| ||A−1||

Here, ||A|| is the matrix norm. There are several norms, e.g. l1-norm for an n×n-matrix||A|| is ||A||1 = max1≤ j≤n ∑

ni=1 |ai j|. κ(A) gauges the transfer of error from the matrix A

and the vector b to the solution x. The rule of thumb is that if κ(A) = 10k, then one canexpect to lose at least k digits of precision in solving the system Ax = b.

A matrix A is called ill-conditioned if the linear system Ax = b is sensitive to perturba-tions in the elements of A, or to perturbations of the components of b. The larger thecondition number is, the more ill-conditioned is the system.

In computational studies, it is extremely important to estimate whether a numerical re-sult can be trusted or not. The condition number provides some evidence regarding thisquestion if one is solving linear systems.

In mathematical software systems such as Matlab, an estimate of the condition number isoften available so that one can estimate the reliability of the numerical result.

Such a criterion may be used for example to choose a suitable solution method.

1.2 Residual vectors

Without having the aid of sophisticated mathematical software, how can we determinewhether a numerical solution to a linear system is correct?

One check is to substitute the obtained numerical solution x back into the original system.This means computing the residual vector

r = Ax−b

The situation is, however, more complicated than one would tend to imagine. A solutionwith a small residual vector is not necessarily a good solution. This can be the case if theoriginal problem is sensitive to roundoff errors - ill-conditioned.

The problem of whether a computed solution is good or not is difficult and mostly beyondthe scope of this course.

7

2 Gaussian elimination with scaled partial pivoting

2.1 Failure of Naive Gaussian elimination

Consider the following simple example{0x1 + x2 = 1x1 + x2 = 2

The Naive Gaussian elimination program would fail here because the pivot element a11 =0.

The program is also unreliable for small values of a11. Consider the system{εx1 + x2 = 1x1 + x2 = 2

where ε is a small number other than zero.

The forward elimination produces the following system{εx1 + x2 = 1

(1−1/ε)x2 = 2−1/ε

In the back substitution, we obtain

x2 =2−1/ε

1−1/εx1 =

1− x2

ε

If ε is small, then 1/ε is large, and in a computer with finite word length, both the values(2−1/ε) and (1−1/ε) would be computed as −1/ε.

Thus the computer calculates that x2 = 1 and x1 = 0. But the correct answer is x2 ≈ 1 andx1 ≈ 1. The relative error in the computed solution for x1 is 100%!!

If we change the order of the equations, the Naive Gaussian algorithm works:{x1 + x2 = 2εx1 + x2 = 1

This becomes {x1 + x2 = 2

(1− ε)x2 = 1−2ε

And the back substitution gives

x2 =1−2ε

1− ε≈ 1 x1 = 2− x2 ≈ 1

This idea of changing the order of the equations is used in a procedure called scaled partialpivoting.

8

2.2 Scaled partial pivoting

We have seen that the order in which we treat the equations significantly affects the accu-racy of the algorithm.

We now describe an algorithm where the order of the equations is determined by thesystem itself. We call the original order in which the equations are given the naturalorder of the system: {1,2,3, . . . ,n}. The new order in which the elimination is performedis denoted by the row vector

l = [l1, l2, . . . , ln]

This is called the index vector. The strategy of interchanging the rows is called scaledpartial pivoting.

We begin by calculating a scale factor for each equation:

si = max1≤ j≤n

|ai j| (1≤ i≤ n)

These n numbers are recorded in a scale vector: s = [s1,s2, . . . ,sn]. The scale vector is notupdated during the procedure because according to a common view, this extra computa-tion is not worth the effort.

Next we choose the pivot equation based on the scale factors. The equation for which theratio |ai1|/si is the greatest is chosen to be the first pivot equation.

Initially we set for the index vector l = [l1, l2, . . . , ln] = [1,2, . . . ,n].

If j is selected to be the index of the first pivot equation, we interchange l j with l1 in theindex vector. Thus we get l = [l j, l2, . . . , l1, . . . , ln].

Next we use the multipliersali1

al11

times the row l1 and subtract these from the equations li for 2 ≤ i ≤ n. (Note that nowl1↔ l j.) This is the first step of the forward elimination.

Note that only the index vector is updated, not the equations!

In the next step, we go through the ratios{|ali2|sl1

: 2≤ i≤ n}

If j is now the index corresponding to the largest ratio, then interchange l j and l2 in theindex vector.

Continue by subtracting the multipliers

ali1

al21

times equation l2 from the remaining equations li for which 3≤ i≤ n.

The same procedure is repeated until n−1 steps have been completed.

9

Numerical exampleConsider the system:

3 −13 9 3−6 4 1 −186 −2 2 4

12 −8 6 10

x1x2x3x4

=

−19−341626

The index vector is initially l = [1,2,3,4].

The scale vector is s = [13,18,6,12].

Step 1.In order to determine the first pivot row, we look at the ratios:{

|ali1|sli

: i = 1,2,3,4}=

{3

13,

618

,66,1212

}≈ {0.23,0.33,1.0,1.0}

The largest value of these ratios occurs first for the index j = 3. So the third row is goingto be our pivot equation in step 1 of the elimination process.

The entries l1 and l j are interchanged in the index vector. Thus we have l = [3,2,1,4].

Now appropriate multiples of the pivot equation (l1 = 3) are subtracted from the otherequations so that zeros are created as coefficients of x1. The result is

0 −12 8 10 2 3 −146 −2 2 40 −4 2 2

x1x2x3x4

=

−27−1816−6

Step 2.In the next step (k = 2), we use the index vector l = [3,2,1,4] and scan the rows l2 = 2,l3 = 1 and l4 = 4 for the largest ratio:{

|ali2|sli

: i = 2,3,4}=

{2

18,1213

,412

}≈ {0.11,0.92,0.33}

The largest is the second value (with index l3 = 1). Thus the first row is our new pivotequation and the index vector becomes l = [3,1,2,4] after interchanging l3 and l2.

Next, multiples of the first equation are subtracted from the two remaining equationsl3 = 2 and l4 = 4. The result is

0 −12 8 10 0 13/3 −83/66 −2 2 40 0 −2/3 5/3

x1x2x3x4

=

−27−45/2

163

10

Step 3.In the third and final step, we first examine the ratios{

|ali3|sli

: i = 3,4}=

{13/318

,2/312

}≈ {0.24,0.06}

The largest value is for l3 = 2, and thus the 2nd row is used as the pivot equation. Theindex vector is unchanged because lk = l j (index of step is the same as the index of pivotequation). We have l = [3,1,2,4].

The remaining task is to subtract the appropriate multiple of equation 2 from equation 4.This gives

0 −12 8 10 0 13/3 −83/66 −2 2 40 0 0 −6/13

x1x2x3x4

=

−27−45/2

16−6/13

Back substitutionThe final index vector gives the order in which the back substitution is performed. Thesolution is obtained by first using l4 = 4 to determine x4, then l3 = 2 to determine x3, thenl2 = 1 to determine x2 and finally l1 = 3 to determine x1.

After the back substitution, we obtain the result

x =

31−21

11

CodeIn the following C implementation, the algorithm is programmed in such a way that it firstcarries out the forward elimination phase on the coefficient array (ai j) only (procedureGauss).

The right-hand-side array (bi) is treated in the next phase (procedure Solve).

Since we are treating the array (bi) later, we need to store the multipliers. These are easilystored in the array (ai j) in positions where 0 entries would have been created otherwise.

Here is the procedure Gauss for forward elimination with scaled partial pivoting.

void Gauss(int n, double a[N][N], int l[N]){

int i, j, k, temp;double aa, r, rmax, smax, xmult;double *s;

/* Allocate memory */s = (double *) malloc((size_t) (n*sizeof(double)));

/* Create the scaling vector */for(i=0; i<n; i++) {

l[i] = i;smax = 0;for(j=0; j<n; j++) {

aa = fabs(a[i][j]);if(aa > smax)

smax = aa;}s[i] = smax;

}

/* Loop over steps */for(k=0; k<n-1; k++) {

/* Choose pivot equation */rmax = 0;j=k;for(i=k; i<n; i++) {

r = fabs(a[l[i]][k]/s[l[i]]);if(r > rmax) {

rmax = r;j = i;

}}/* Interchange indeces */temp = l[j];l[j] = l[k];l[k] = temp;/* Elimination for step k */for(i=k+1; i<n; i++) {

xmult = a[l[i]][k]/a[l[k]][k];a[l[i]][k] = xmult;for(j=k+1; j<n; j++) {

a[l[i]][j] = a[l[i]][j] - xmult*a[l[k]][j];}

}}free(s);

}

12

And here is the procedure Solve for determining the array (bi) and for the back substi-tution phase:

void Solve(int n, double a[N][N], int l[N], double b[N], double x[N]){

int i, j, k;double sum;

/* Determine array bi *//* Elements a[l[i]][k] are the multipliers */for(k=0; k<n-1; k++) {

for(i=k+1; i<n; i++) {b[l[i]] = b[l[i]]-a[l[i]][k]*b[l[k]];

}}/* Back substitution phase */x[n-1]= b[l[n-1]]/a[l[n-1]][n-1];for(i=n-2; i>=0; i--) {

sum = b[l[i]];for(j=i+1; j<n; j++) {

sum = sum - a[l[i]][j]*x[j];}x[i] = sum/a[l[i]][i];

}}

If we use the program on the same problem as solved before by hand and using the naiveGaussian elimination program, we get the following results (note that in C the indexinggoes from 0 to n−1, here it has been changed to run from 1 to n:

Scale vector:s[1] = 6.000000s[2] = 12.000000s[3] = 13.000000s[4] = 18.000000

Final index vector:l[1] = 1l[2] = 3l[3] = 4l[4] = 2

Arrays (aij) and (bi) (multipliers in place of 0’s):ai1 ai2 ai3 ai4 | bi

6.0000 -2.0000 2.0000 4.0000 | 16.00002.0000 0.3333 -0.1538 | -0.4615 | -0.46150.5000 | -12.0000 8.0000 1.0000 | -27.0000-1.0000 -0.1667 | 4.3333 -13.8333 | -22.5000

Solution:x[1] = 3.000000x[2] = 1.000000x[3] = -2.000000x[4] = 1.000000

We obtain the same solution as before but the arrays (ai j) and (bi) are different (due todifferent sequence of pivot equations).

13

2.3 Long operation count

Solving large systems of linear equations on a computer can be expensive. We will nowcount the number of long operations involved in the Gaussian elimination with partialpivoting.

By long operations we mean multiplications and divisions. We make no distinction be-tween the two, although division is more time consuming than multiplication.

i. Procedure Gauss

Step 1:- Choice of pivot element: n divisions- Multipliers (one for each row): n−1 divisions- New elements: n−1 multiplications / row→ (n−1)× (n−1)- Total: n+(n−1)+(n−1)(n−1) = n+n(n−1) = n2 long operations

Step 2:- Involves n−1 rows and n−1 columns- Total: (n−1)2 long operations

Whole procedure:-Total:

n2 +(n−1)2 +(n−2)2 + . . .+32 +22 =n6(n+1)(2n+1)−1≈ n3

3

ii. Procedure Solve

Forward processing of (bi): - Involves n−1 steps- First step: n−1 multiplications- Second step: n−2 multiplications, etc.- Total: (n−1)+(n−2)+ . . .+3+2+1 = n

2(n−1)

Back substitution: - One operation in first step, two in the second and so on.- Total: 1+2+3+ . . .+n = n

2(n+1)

Whole procedure: -Total:

n2(n−1)+

n2(n+1) =

n2(n−1)(2n) = n2

Theorem on long operations.The forward elimination phase of the Gaussian elimination algorithm with scaled partialpivoting, if applied only to the n×n coefficient array, involves approximately n3/3 longoperations (multiplications or divisions). Solving for x requires an additional n2 longoperations.

14

3 LU factorization

An n×n system of linear equations can be written in matrix form

Ax = b

where the coefficient matrix A has the form

A =

a11 a12 a13 . . . a1na21 a22 a23 . . . a2na31 a32 a33 . . . a3n

......

... . . . ...an1 an2 an3 . . . ann

We now want to show that the naive Gaussian algorithm applied to A yields a factorizationof A into two products:

A = LU

where L has a lower triangular form:

L =

1

l21 1l31 l32 1...

...... . . .

ln1 ln2 ln3 . . . 1

and U has a upper triangular form:

U =

u11 u12 u13 . . . u1n

u22 u23 . . . u2nu33 . . . u3n

. . . ...unn

This is called the LU factorization.

15

3.1 Benefit of LU factorization

Using the LU factorization of A we can write

Ax = (LU)x = L(Ux) = b

This means that we can solve the system of linear equations in two parts:

L z = bU x = z

Solving these sets of equations is trivial since the matrices are in triangular form.

We first solve for z using the first equation. Since L is lower triangular, we can useforward substitution:

z1 =b1

l11

zi =1lii

[bi−

i−1

∑j=1

li jz j

](i = 2,3, . . . ,n)

Once we have z, the second equation can be solved using back substitution:

xn =zn

unn

xi =1uii

[zi−

N

∑j=i+1

ui jx j

](i = n−1,n−2, . . . ,1)

The benefit of this method is that once we have the LU decomposition of A, we can usethe same matrices U and L to solve for several different vectors b.

The CPU time requirement for the above forward and back substitutions is O(n2), whereasthe Gauss elimination algorithm is O(n3).

16

Numerical exampleThe first example (in Naive Gaussian elimination) can be written in matrix form as:

6 −2 2 412 −8 6 103 −13 9 3−6 4 1 −18

x1x2x3x4

=

1626−19−34

We obtained the following upper triangular form for the system as a result of the forwardelimination phase:

6 −2 2 40 −4 2 20 0 2 −50 0 0 −3

x1x2x3x4

=

16−6−9−3

The forward elimination phase can be interpreted as starting from the first equation, Ax =b, and proceeding to

MAx = Mb

where M is a matrix chosen so that MA is the coefficient matrix in upper triangular form:

MA =

6 −2 2 40 −4 2 20 0 2 −50 0 0 −3

≡ U

The forward elimination phase of the naive Gaussian elimination algorithm consists of aseries of steps. The first step gives us the following system:

6 −2 2 40 −4 2 20 −12 8 10 2 3 −14

x1x2x3x4

=

16−6−27−18

The first step can be accomplished by multiplying by a lower triangular matrix M1:

M1Ax = M1b

where

M1 =

1 0 0 0−2 1 0 0−1

2 0 1 01 0 0 1

Notice the special form of this matrix: the diagonal elements are all 1’s and the only othernonzero elements are in the first column. These are the negatives of the multipliers.

17

The second step results in the system:6 −2 2 40 −4 2 20 0 2 −50 0 4 −13

x1x2x3x4

=

16−6−9−21

which is equivalent to

M2M1Ax = M2M1b

where

M2 =

1 0 0 00 1 0 00 −3 1 00 1

2 0 1

Finally, step 3 gives the system in the upper triangular form

6 −2 2 40 −4 2 20 0 2 −50 0 0 −3

x1x2x3x4

=

16−6−9−3

which is equivalent to

M3M2M1Ax = M3M2M1b

where

M3 =

1 0 0 00 1 0 00 0 1 00 0 −2 1

Thus we obtain the matrix M as a product of the three multiplier matrices:

M = M3M2M1

18

We can now derive a different interpretation of the forward elimination phase. We seethat

A = M−1U

= M−11 M−1

2 M−13 U

= LU

The inverse of the Mk’s is obtained easily by changing the signs of the multipliers. Thuswe have

L =

1 0 0 02 1 0 012 0 1 0−1 0 0 1

1 0 0 00 1 0 00 3 1 00 −1

2 0 1

1 0 0 00 1 0 00 0 1 00 0 2 1

=

1 0 0 02 1 0 012 3 1 0−1 −1

2 2 1

This is a unit lower triangular matrix composed of the multipliers!

We can easily verify that

LU =

1 0 0 02 1 0 012 3 1 0−1 −1

2 2 1

6 −2 2 40 −4 2 20 0 2 −50 0 0 −3

=

6 −2 2 412 −8 6 103 −13 9 3−6 4 1 −18

= A

In LU factorization, the matrix A is factored or decomposed into a unit lower triangularmatrix L and an upper triangular matrix U.

• The lower triangular elements of L are the multipliers located in the positions ofthe elements they annihilated from A.

• U is the final coefficient matrix obtained after the forward elimination phase.

Note that the naive Gaussian elimination algorithm replaces the original coefficient matrixA with its LU factorization.

19

3.2 LU factorization theorem

Theorem.Let A = (ai j) be an n×n matrix. Assume that the forward elimination phase of the naiveGaussian algorithm is applied to A without encountering any 0 divisors. Let the resultingmatrix be denoted by A = (ai j). If

L =

1 0 0 . . . 0

a21 1 0 . . . 0a31 a32 1 . . . 0

...... . . . . . . ...

an1 an2 . . . an,n−1 1

and

U =

a11 a12 a13 . . . a1n0 a22 a23 . . . a2n0 0 a33 . . . a3n...

... . . . . . . ...0 0 . . . 0 an,n

then A = LU.

Notice that we rely here on not encountering any zero divisors in the algorithm.

20

3.3 Solving linear systems using LU factorization

Once the LU factorization of A is available, we can solve the system

A x = b

by writingL U x = b

Then we solve two triangular systems:

L z = b

for z andU x = z

for x.

This is particularly useful if we have a problem that involves the same coefficient matrixA and several different right-hand side vectors b.

z is obtained by the pseudocode (forward substitution):

real array (bi)n, (li j)n×n, (zi)ninteger i, nz1← b1for i = 2 to n do

zi← bi−∑i−1j=1 li jz j

end for

And similarly x is obtained by the pseudocode (back substitution):

real array (xi)n, (ui j)n×n, (zi)ninteger i, nxn← zn/unnfor i = n−1 to 1 step-1 do

xi← 1uii

(zi−∑

nj=i+1 ui jx j

)end for

21

3.4 Software packages

Matlab and Maple produce factorizations of the form

P A = L U

where P is a permutation matrix corresponding to the pivoting strategy used. Exampleof such a matrix is

P =

0 1 0 00 0 1 00 0 0 11 0 0 0

A permutation matrix is an n× n matrix which is obtained from the unit matrix by per-muting its rows. Permuting the rows of any n× n matrix A can be accomplished bymultiplying A on the left by P.

When Gaussian elimination with row pivoting is performed on A, the result is expressibleas P A = L U.

The matrix P A is A with rows rearranged.

We can use the LU factorization of P A to solve the system A x = b in the following way:

P A x = P bL U x = P b

L y = P b

This is easily solved for y. The solution is obtained by solving

U x = y

for x.

22