SE301: Numerical Methods Topic 3: Solution of Systems of Linear Equations Lectures 12-17:

CISE301_Topic3 1

SE301: Numerical Methods

Topic 3: Solution of Systems of Linear

Equations Lectures 12-17:

KFUPM

Read Chapter 9 of the textbook

CISE301_Topic3 2

Lecture 12Vector, Matrices, and

Linear Equations

CISE301_Topic3 3

VECTORSVECTORS

1

0

0

0

,

0

1

0

0

,

0

0

1

0

,

0

0

0

1

1

2tor column vec 241 vector row

:Examples

numbers ofarray ldimensiona one a:Vector

4321 eeeevectorsIdentity

CISE301_Topic3 4

MATRICESMATRICES

1200

1410

0143

0021

lTridiagona,

6000

0000

0040

0001

diagonal

10

01matrix identity

0

0

00

00matrix zero

:Examples

numbers ofarray ldimensiona twoa:Matrix

CISE301_Topic3 5

MATRICESMATRICES

1000

1400

0140

3121

ngular upper tria,

451

501

112

symmetric

:Examples

CISE301_Topic3 6

Determinant of a Determinant of a MATRICESMATRICES

82)015(1)512(1)25(2

50

1-31-

45

1-31-

45

502

451

501

132

det

:Examples

only matrices squarefor Defined

CISE301_Topic3 7

Adding and Multiplying Adding and Multiplying MatricesMatrices

ji

ji

m

k

,bacBAC *

pm ifonly defined is ABCproduct The *

q)B(p and m)(nA matrices twooftion Multiplica

,bacBAC *

size same thehave they ifonly Defined *

B andA matrices twoofaddition The

1kjikij

ijijij

CISE301_Topic3 8

Systems of Linear Equations

formMatrix form Standard

7

5

3

601

315.2

342

76

535.2

3342

formsdifferent in

presented becan equationslinear of systemA

3

2

1

31

321

321

x

x

x

xx

xxx

xxx

CISE301_Topic3 9

Solutions of Linear Equations

52

3

:equations following theosolution t a is 2

1

21

21

2

1

xx

xx

x

x

CISE301_Topic3 10

Solutions of Linear Equations A set of equations is inconsistent if there

exists no solution to the system of equations:

ntinconsiste are equations These

542

32

21

21

xx

xx

CISE301_Topic3 11

Solutions of Linear Equations Some systems of equations may have infinite

number of solutions

allfor solution ais)3(5.0

solutions ofnumber infinite have

642

32

2

1

21

21

aa

a

x

x

xx

xx

CISE301_Topic3 12

Graphical Solution of Systems ofLinear Equations

52

3

21

21

xx

xx

Solution

x1=1, x2=2

CISE301_Topic3 13

Cramer’s Rule is Not Practical

way efficient in computed are tsdeterminan theif used be canIt

needed. are tionsmultiplica 102.38 system, 30by 30a solve To

tions.multiplica 1)N!-1)(N(N requires system Nby N solve To

. systems largefor practicalnot is RulesCramer'

2

21

1151

31

,1

21

1125

13

system thesolve toused be can RulesCramer'

35

21

xx

CISE301_Topic3 14

Naive Gaussian Elimination Examples

Lecture 13 Naive Gaussian

Elimination

CISE301_Topic3 15

Naive Gaussian Elimination The method consists of two steps:

Forward Elimination: the system is reduced to upper triangular form. A sequence of elementary operations is used.

Backward Substitution: Solve the system starting from the last variable.

'

'

'00

''0

3

2

1

3

2

1

33

2322

131211

3

2

1

3

2

1

333231

232221

131211

b

b

b

x

x

x

a

aa

aaa

b

b

b

x

x

x

aaa

aaa

aaa

CISE301_Topic3 16

Elementary Row Operations

Adding a multiple of one row to another

Multiply any row by a non-zero constant

CISE301_Topic3 17

ExampleForward Elimination

18

27

6

16

14320

18120

2240

4226

4 3, 2,equationsfrom Eliminate:Step1

nEliminatio Forward:1Part

34

19

26

16

18146

39133

106812

4226

4

3

2

1

1

4

3

2

1

x

x

x

x

x

x

x

x

x

CISE301_Topic3 18


3

9

6

16

3000

5200

2240

4226

4equation from Eliminate:Step3

21

9

6

16

13400

5200

2240

4226

4 ,3equationsfrom Eliminate:Step2

4

3

2

1

3

4

3

2

1

2

x

x

x

x

x

x

x

x

x

x

CISE301_Topic3 19


3

9

6

16

3000

5200

2240

4226

34

19

26

16

18146

39133

106812

4226

:nEliminatio Forward theofSummary

4

3

2

1

4

3

2

1

x

x

x

x

x

x

x

x

CISE301_Topic3 20

ExampleBackward Substitution

36

)1(4)2(2)1(216,1

4

)1(2)2(26

22

59,1

3

3

for solve ,...for solvethen ,for Solve

3

9

6

16

3000

5200

2240

4226

12

34

134

4

3

2

1

xx

xx

xxx

x

x

x

x

CISE301_Topic3 21

Forward Elimination

ni

ba

abb

njaa

aaa

x

ni

ba

abb

njaa

aaa

x

iii

ji

ijij

iii

ji

ijij

3

)2(

eliminate To

2

)1(

eliminate To

222

2

222

2

2

111

1

111

1

1

CISE301_Topic3 22

Forward Elimination

.eliminated is until Continue

1

)(

eliminate To

1

n

kkk

ikii

kjkk

ikijij

k

x

nik

ba

abb

njkaa

aaa

x

CISE301_Topic3 23

Backward Substitution

ii

n

ijjjii

i

nn

nnnnnnnn

nn

nnnnn

nn

nn

a

xab

x

a

xaxabx

a

xabx

a

bx

,

1,

2,2

11,2,222

1,1

,111

,

CISE301_Topic3 24

Summary of the Naive Gaussian Elimination Example Problems with Naive Gaussian Elimination

Failure due to zero pivot element Error

Pseudo-Code

Lecture 14Naive Gaussian

Elimination

CISE301_Topic3 25

Naive Gaussian Elimination

o The method consists of two stepso Forward Elimination: the system is reduced to

upper triangular form. A sequence of elementary operations is used.

o Backward Substitution: Solve the system starting from the last variable. Solve for xn ,xn-1,…x1.

'

'

'00

''0

3

2

1

3

2

1

33

2322

131211

3

2

1

3

2

1

333231

232221

131211

b

b

b

x

x

x

a

aa

aaa

b

b

b

x

x

x

aaa

aaa

aaa

CISE301_Topic3 26

Example 1

1775

64

832

11

333723

11

22210232

)(1832

3 ,2equations from Eliminate:Step1___nEliminatio Forward:1Part

:nEliminatioGaussian Naive using Solve

32

32

321

321

321

321

1

xx

xx

xxx

eqeqeqxxx

eqeqeqxxx

equationpivotunchangedeqxxx

x

CISE301_Topic3 27

Example 1

1313

64

832

21

5331775

)(264

1832

3equationfrom Eliminate:Step2nEliminatio Forward:1Part

3

32

321

32

32

321

2

x

xx

xxx

eqeqeqxx

equationpivotunchangedeqxx

unchangedeqxxx

x

CISE301_Topic3 28

Example 1Backward Substitution

1

2

1

issolution The

1328

21

46

113

13

3

2

1

1,1

32

1,1

33,122,111

3

2,2

33,222

3,3

33

x

x

x

a

xx

a

xaxabx

x

a

xabx

a

bx

CISE301_Topic3 29

Determinant

13detdet

1300

410

321

A'

213

232

321

A

:Example

tdeterminan affect thenot do operations elementary The

operations Elementary

(A')(A)

CISE301_Topic3 30

How Many Solutions Does a System of Equations AX=B Have?

0 elements0 elements

B ingcorrespondB ingcorrespond

rows zerorows zero

moreor one has moreor one hasrows zero no has

matrix reducedmatrix reducedmatrix reduced

0det(A)0det(A)0det(A)

Infinitesolution NoUnique

CISE301_Topic3 31

Examples

5.1

!105.0

0

#:

0

2

00

21

1

2

00

21

1

1

20

21

4

2

42

21

3

2

42

21

2

1

43

21

solutions of # infintesolution NoUnique

XimpossibleX

solutionsInfinitesolutionNosolution

XXX

XXX

Pseudo-Code: Forward EliminationDo k = 1 to n-1

Do i = k+1 to nfactor = ai,k / ak,k

Do j = k+1 to nai,j = ai,j – factor * ak,j

End Dobi = bi – factor * bk

End DoEnd DoCISE301_Topic3 32

Pseudo-Code: Back Substitutionxn = bn / an,n

Do i = n-1 downto 1sum = bi

Do j = i+1 to nsum = sum – ai,j * xj

End Doxi = sum / ai,i

End Do

CISE301_Topic3 33

CISE301_Topic3 34

Lectures 15-16:Gaussian Elimination

with Scaled Partial Pivoting

Problems with Naive Gaussian Elimination Definitions and Initial step Forward Elimination Backward substitution Example

CISE301_Topic3 35

Problems with Naive Gaussian Elimination

o The Naive Gaussian Elimination may fail for very simple cases. (The pivoting element is zero).

o Very small pivoting element may result in serious computation errors

2

1

11

10

2

1

x

x

2

1

11

110

2

110

x

x

CISE301_Topic3 36

Example 2

1

1

1

1

3524

3685

4123

1211

: Pivoting PartialScaled with

on EliminatiGaussian using system following theSolve

4

3

2

1

x

x

x

x

CISE301_Topic3 37

Example 2Initialization step

4321LVectorIndex

5842S vectorScale

1

1

1

1

3524

3685

4123

1211

4

3

2

1

x

x

x

xScale vector:

disregard sign

find largest in magnitude in each row

CISE301_Topic3 38

Why Index Vector? Index vectors are used because it is much

easier to exchange a single index element compared to exchanging the values of a complete row.

In practical problems with very large N, exchanging the contents of rows may not be practical.

CISE301_Topic3 39

Example 2Forward Elimination-- Step 1: eliminate x1

]1324[

Exchangeequation pivot first theis 4equation

toscorrespondmax 5

4,

8

5,

4

3,

2

14,3,2,1

]4321[

]5842[

1

1

1

1

3524

3685

4123

1211

equationpivot theofSelection

14

4

1,

4

3

2

1

L

landl

liS

aRatios

L

S

x

x

x

x

i

i

l

l

CISE301_Topic3 40


1

25.2

75.1

25.1

3524

75.025.05.50

75.175.25.00

25.075.05.10

1

1

1

1

3524

3685

4123

1211

B andA Update

4

3

2

1

4

3

2

1

x

x

x

x

x

x

x

x

First pivot equation

CISE301_Topic3 41


]2314[2

5.1

8

5.5

4

5.04,3,2:Ratios

]1324[]5842[

1

25.2

75.1

25.1

3524

75.025.05.50

75.175.25.00

25.075.05.10

equationpivot second theofSelection

2,

4

3

2

1

LiS

a

LS

x

x

x

x

i

i

l

l

CISE301_Topic3 42


1

9

1667.2

25.1

3524

2000

8333.15.200

25.075.05.10

]3214[

1

8333.6

1667.2

25.1

3524

6667.125.000

8333.15.200

25.075.05.10

4

3

2

1

4

3

2

1

x

x

x

x

L

x

x

x

xThird pivot equation

CISE301_Topic3 43


7.23334

2531

1.13335.1

75.025.025.1

2.43275.28333.11667.2

,5.429

]3214[

1

9

1667.2

25.1

3524

2000

8333.15.200

25.075.05.10

234

1,

22,433,444,441

34

2,1

33,144,112

4

3,2

44,223

4,3

34

4

3

2

1

1

xxx

a

xaxaxabx

xx

a

xaxabx

xa

xabx

a

bx

L

x

x

x

x

l

CISE301_Topic3 44

Example 3

1

1

1

1

3524

3685

4123

1211

Pivoting Partial Scaledwith

nEliminatioGaussian using sytstem following theSolve

4

3

2

1

x

x

x

x

CISE301_Topic3 45

Example 3Initialization step

4321LVectorIndex

5842S vectorScale

1

1

1

1

3524

3685

4123

1211

4

3

2

1

x

x

x

x

CISE301_Topic3 46


]1324[

Exchangeequation pivot first theis 4equation

toscorrespondmax 5

4,

8

5,

4

3,

2

14,3,2,1

]4321[

]5842[

1

1

1

1

3524

3685

4123

1211

equationpivot theofSelection

14

41,

4

3

2

1

L

landl

liS

aRatios

L

S

x

x

x

x

i

i

l

l

CISE301_Topic3 47


1

25.2

75.1

25.1

3524

75.025.05.100

75.175.25.00

25.075.05.10

1

1

1

1

3524

3685

4133

1211

B andA Update

4

3

2

1

4

3

2

1

x

x

x

x

x

x

x

x

CISE301_Topic3 48


]1234[2

5.1

8

5.10

4

5.04,3,2:Ratios

]1324[]5842[

1

25.2

75.1

25.1

3524

75.025.05.100

75.175.25.00

25.075.05.10

equationpivot second theofSelection

2,

4

3

2

1

LiS

a

LS

x

x

x

x

i

i

l

l

CISE301_Topic3 49


1

2.25

1.8571

0.9286

3524

75.025.05.100

1.71432.7619-00

0.35710.785700

]2314[

1

25.2

75.1

25.1

3524

75.025.05.100

75.175.25.00

25.075.05.10

B andA Updating

4

3

2

1

4

3

2

1

x

x

x

x

L

x

x

x

x

CISE301_Topic3 50


]1234[2

0.7857

4

2.76194,3:Ratios

]1234[]5842[

1

2.25

1.8571

0.9286

3524

75.025.05.100

1.71432.761900

0.35710.785700

equationpivot third theofSelection

3,

4

3

2

1

LiS

a

LS

x

x

x

x

i

i

l

l

CISE301_Topic3 51


1

2.25

1.8571

1.4569

3524

75.025.05.100

1.71432.761900

0.8448000

]1234[

1

2.25

1.8571

0.9286

3524

75.025.05.100

1.71432.761900

0.35710.785700

4

3

2

1

4

3

2

1

x

x

x

x

L

x

x

x

x

CISE301_Topic3 52


1.86734

2531

0.3469

0.39802.7619

1.71431.8571,1.7245

0.84481.4569

]1234[

1

2.25

1.8571

1.4569

3524

75.025.05.100

1.71432.761900

0.8448000

234

1,

22,33,44,1

2,

33,44,2

4

3,

44,3

4,4

4

3

2

1

1

1111

2

222

3

33

4

4

xxx

a

xaxaxabx

a

xaxabx

xa

xabx

a

bx

L

x

x

x

x

l

llll

l

lll

l

ll

l

l

CISE301_Topic3 53

How Do We Know If a Solution is Good or Not

Given AX=BX is a solution if AX-B=0

Compute the residual vector R= AX-B

Due to rounding error, R may not be zero

ii

rmax if acceptable issolution The

CISE301_Topic3 54

How Good is the Solution?

0.001

0.003

0.002

0.005

:Residues

1.7245

0.3980

0.3469

1.8673

1

1

1

1

3524

3685

4123

1211

4

3

2

1

4

3

2

1

R

x

x

x

x

solution

x

x

x

x

CISE301_Topic3 55

Remarks: We use index vector to avoid the need to move

the rows which may not be practical for large problems.

If we order the equation as in the last value of the index vector, we have a triangular form.

Scale vector is formed by taking maximum in magnitude in each row.

Scale vector does not change. The original matrices A and B are used in

checking the residuals.

CISE301_Topic3 56

Lecture 17 Tridiagonal & Banded

Systems and Gauss-Jordan

Method

Tridiagonal Systems Diagonal Dominance Tridiagonal Algorithm Examples Gauss-Jordan Algorithm

CISE301_Topic3 57

Tridiagonal Systems: The non-zero elements are

in the main diagonal, super diagonal and subdiagonal.

aij=0 if |i-j| > 1

5

4

3

2

1

5

4

3

2

1

61000

14100

02620

00143

00015

b

b

b

b

b

x

x

x

x

x

Tridiagonal Systems

CISE301_Topic3 58

Occur in many applications Needs less storage (4n-2 compared to n2 +n for the general cases)

Selection of pivoting rows is unnecessary (under some conditions)

Efficiently solved by Gaussian elimination

Tridiagonal Systems

CISE301_Topic3 59

Based on Naive Gaussian elimination. As in previous Gaussian elimination algorithms

Forward elimination step Backward substitution step

Elements in the super diagonal are not affected. Elements in the main diagonal, and B need

updating

Algorithm to Solve Tridiagonal Systems

CISE301_Topic3 60

Tridiagonal System

'

'3

'2

1

3

2

1

'1

'3

2'2

11

3

2

1

3

2

1

1

1

32

221

11

updatednot are elements The

elements and theupdate toneed zeros, be willelements theAll

nnn

n

nnnn

n

b

b

b

b

x

x

x

x

d

c

d

cd

cd

b

b

b

b

x

x

x

x

da

c

da

cda

cd

c

bda

CISE301_Topic3 61

Diagonal Dominance

row. ingcorrespond in the elements of sum the

nlarger tha iselement diagonaleach of magnitude The

)1(aa

ifdominant diagonally is matrix A

,1ijii nifor

An

ijj

CISE301_Topic3 62

Diagonal Dominance

dominant DiagonallyNot dominant Diagonally

121

232

103

521

161

103

:Examples

CISE301_Topic3 63

Diagonally Dominant Tridiagonal System

A tridiagonal system is diagonally dominant if

Forward Elimination preserves diagonal dominance

)1(1 niacd iii

CISE301_Topic3 64

Solving Tridiagonal System

1,...,2,1for 1

onSubstituti Backward

2

nEliminatio Forward

1

11

1

11

1

nnixcbd

x

d

bx

nibd

abb

cd

add

iiii

i

n

nn

ii

iii

ii

iii

CISE301_Topic3 65

Example

1,2,3for 1

,

onSubstituti Backward

42,

nEliminatio Forward

6

8

9

12

,2

2

2

,1

1

1

,

5

5

5

5

6

8

9

12

51

251

251

25

Solve

1

11

11

1

1

4

3

2

1

ixcbd

xd

bx

ibd

abbc

d

add

BCAD

x

x

x

x

iiii

in

nn

ii

iiii

i

iii

CISE301_Topic3 66

Example

5619.45652.4

5652.616,5619.4

5652.4

215

5652.66.4

6.618,5652.4

6.4

215

6.65

1219,6.4

5

215

nEliminatio Forward

6

8

9

12

,2

2

2

,1

1

1

,

5

5

5

5

33

3443

3

344

22

2332

2

233

11

1221

1

122

bd

abbc

d

add

bd

abbc

d

add

bd

abbc

d

add

BCAD

CISE301_Topic3 67

ExampleBackward Substitution

After the Forward Elimination:

Backward Substitution:

25

1212

16.4

126.6

15652.4

125652.6

,15619.4

5619.4

5619.45652.66.612,5619.45652.46.45

1

2111

2

3222

3

4333

4

44

d

xcbx

d

xcbx

d

xcbx

d

bx

BD TT

CISE301_Topic3 68

Gauss-Jordan Method The method reduces the general system of

equations AX=B to IX=B where I is an identity matrix.

Only Forward elimination is done and no backward substitution is needed.

It has the same problems as Naive Gaussian elimination and can be modified to do partial scaled pivoting.

It takes 50% more time than Naive Gaussian method.

CISE301_Topic3 69

Gauss-Jordan MethodExample

2

7

0

200

560

111

11

233

11

422

2/11

32fromxEleminate1

2

7

0

422

124

222

3

2

1

1

3

2

1

x

x

x

eqeqeq

eqeqeq

eqeq

andequationsStep

x

x

x

CISE301_Topic3 70


2

1.1667

1.1667

200

0.833310

1667.001

21

033

21

111

6/22

31fromxEleminate2

2

7

0

200

560

111

3

2

1

2

3

2

1

x

x

x

eqeqeq

eqeqeq

eqeq

andequationsStep

x

x

x

CISE301_Topic3 71


1

2

1

100

010

001

31

0.833322

31

1667.011

2/33

21fromxEleminate3

2

1.1667

1.1667

200

0.833310

1667.001

3

2

1

3

3

2

1

x

x

x

eqeqeq

eqeqeq

eqeq

andequationsStep

x

x

x

CISE301_Topic3 72


1

2

1

1

2

1

100

010

001

2

7

0

422

124

222

3

2

1

3

2

1

3

2

1

x

x

x

issolution

x

x

x

todtransformeis

x

x

x

SE301: Numerical Methods Topic 3: Solution of Systems of Linear Equations Lectures 12-17:

Documents

Transcript of SE301: Numerical Methods Topic 3: Solution of Systems of Linear Equations Lectures 12-17: