Steepest Decent and Conjugate Gradients (CG)
description
Transcript of Steepest Decent and Conjugate Gradients (CG)
Steepest Decent and Conjugate Gradients (CG)
Steepest Decent and Conjugate Gradients (CG)
• Solving of the linear equation system bAx
Steepest Decent and Conjugate Gradients (CG)
• Solving of the linear equation system • Problem: dimension n too big, or not enough time for
gauss eliminationIterative methods are used to get an
approximate solution.
bAx
Steepest Decent and Conjugate Gradients (CG)
• Solving of the linear equation system • Problem: dimension n too big, or not enough time for
gauss eliminationIterative methods are used to get an
approximate solution.• Definition Iterative method: given starting point , do
steps hopefully converge to the right solution
bAx
0x,, 21 xx
x
starting issues
starting issues
• Solving is equivalent to minimizing bAx cxbAxxxf TT
21:)(
starting issues
• Solving is equivalent to minimizing
• A has to be symmetric positive definite:
bAx cxbAxxxf TT
21:)(
00 xAxxAA TT
starting issues
• 021
21)(
! bAxbAxxAxf
symmetricAT
starting issues
•
• If A is also positive definite the solution of is the minimum
021
21)(
! bAxbAxxAxf
symmetricAT
bAx
starting issues
•
• If A is also positive definite the solution of is the minimum
021
21)(
! bAxbAxxAxf
symmetricAT
bAx
00
11
21
21)(
d
TT AddcbAbdbAf
starting issues
• error: The norm of the error shows how far we are away from the exact solution, but can’t be computed without knowing of the exact solution .
xxe ii :
x
starting issues
• error: The norm of the error shows how far we are away from the exact solution, but can’t be computed without knowing of the exact solution .
• residual: can be calculated
xxe ii :
x)(: xfAeAxbr iii
Steepest Decent
Steepest Decent
• We are at the point . How do we reach ?ix 1ix
Steepest Decent
• We are at the point . How do we reach ?• Idea: go into the direction in which decreases most
quickly ( )
ix 1ix)(xf
ii rxf )(
Steepest Decent
• We are at the point . How do we reach ?• Idea: go into the direction in which decreases most
quickly ( )• how far should we go?
ix 1ix)(xf
ii rxf )(
Steepest Decent
• We are at the point . How do we reach ?• Idea: go into the direction in which decreases most
quickly ( )• how far should we go?
Choose so that is minimized:
ix 1ix)(xf
ii rxf )(
)( ii rxf
Steepest Decent
• We are at the point . How do we reach ?• Idea: go into the direction in which decreases most
quickly ( )• how far should we go?
Choose so that is minimized:
ix 1ix)(xf
ii rxf )(
)( ii rxf
0)( ii rxfdd
Steepest Decent
• We are at the point . How do we reach ?• Idea: go into the direction in which decreases most
quickly ( )• how far should we go?
Choose so that is minimized:
ix 1ix)(xf
ii rxf )(
)( ii rxf
0)( ii rxfdd
0)( iT
ii rrxf
Steepest Decent
• We are at the point . How do we reach ?• Idea: go into the direction in which decreases most
quickly ( )• how far should we go?
Choose so that is minimized:
ix 1ix)(xf
ii rxf )(
)( ii rxf
0)( ii rxfdd
0)( iT
ii rrxf
0))(( iT
ii rbrxA
Steepest Decent
• We are at the point . How do we reach ?• Idea: go into the direction in which decreases most
quickly ( )• how far should we go?
Choose so that is minimized:
ix 1ix)(xf
ii rxf )(
)( ii rxf
0)( ii rxfdd
0)( iT
ii rrxf
0))(( iT
ii rbrxA
iT
r
iiT
i rAxbrAri
)()(
Steepest Decent
• We are at the point . How do we reach ?• Idea: go into the direction in which decreases most
quickly ( )• how far should we go?
Choose so that is minimized:
ix 1ix)(xf
ii rxf )(
)( ii rxf
0)( ii rxfdd
0)( iT
ii rrxf
0))(( iT
ii rbrxA
iT
r
iiT
i rAxbrAri
)()( i
Ti
iTi
Arrrr
Steepest Decent
one step of steepest decent can be calculated as follows:
iiii
Ti
iTi
i
ii
rxxArrrr
Axbr
1
Steepest Decent
one step of steepest decent can be calculated as follows:
• stopping criterion: or with an given small It would be better to use the error instead of the residual, but you can’t calculate the error.
iiii
Ti
iTi
i
ii
rxxArrrr
Axbr
1
maxii 0rri
Steepest Decent
Method of steepest decent:
1
)(
0
00max
0
0
iiAxbrrxx
Arrrr
rrrrandiiwhile
rrAxbr
i
T
T
TT
Steepest Decent
• As you can see the starting point is important!
Steepest Decent
• As you can see the starting point is important!
When you know anything about the solution use it to guess a good starting point. Otherwise you can choose a starting point you want e.g. .00 x
Steepest Decent - Convergence
Steepest Decent - Convergence
• Definition energy norm: Axxx TA:
Steepest Decent - Convergence
• Definition energy norm:
• Definition condition:
( is the largest and the smallest eigenvalue of A)
Axxx TA:
min
max:
max min
Steepest Decent - Convergence
• Definition energy norm:
• Definition condition:
( is the largest and the smallest eigenvalue of A)
•
convergence gets worse when the condition gets larger
Axxx TA:
min
max:
max min
A
i
Aiee 01
1
Conjugate Gradients
Conjugate Gradients
• is there a better direction?
Conjugate Gradients
• is there a better direction?• Idea: orthogonal search directions110 ,,, nddd
Conjugate Gradients
• is there a better direction?• Idea: orthogonal search directions110 ,,, nddd
1
0
n
iiidx
Conjugate Gradients
• is there a better direction?• Idea: orthogonal search directions
• only walk once in each direction and minimize
110 ,,, nddd
1
0
n
iiidx
Conjugate Gradients
• is there a better direction?• Idea: orthogonal search directions
• only walk once in each direction and minimizemaximal n steps are needed to reach the exact
solution
110 ,,, nddd
1
0
n
iiidx
Conjugate Gradients
• is there a better direction?• Idea: orthogonal search directions
• only walk once in each direction and minimizemaximal n steps are needed to reach the exact
solution has to be orthogonal to
110 ,,, nddd
1
0
n
iiidx
1 ie id
Conjugate Gradients
• example with the coordinate axes as orthogonal search directions:
Conjugate Gradients
• example with the coordinate axes as orthogonal search directions:
Problem: can’t be computed
because
(you don’t know !)
iTi
iTi
i dded
ie
Conjugate Gradients
• new idea: A-orthogonal110 ,,, nddd
Conjugate Gradients
• new idea: A-orthogonal• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
Conjugate Gradients
• new idea: A-orthogonal• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
• now has to be A-orthogonal to
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
1ie id
Conjugate Gradients
• new idea: A-orthogonal• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
• now has to be A-orthogonal to
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
1ie id
iTi
iTi
iTi
iTi
i Addrd
AddAed
Conjugate Gradients
• new idea: A-orthogonal• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
• now has to be A-orthogonal to
can be computed!
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
1ie id
iTi
iTi
iTi
iTi
i Addrd
AddAed
Conjugate Gradients
• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).
iu
Conjugate Gradients
• Gram-Schmidt: linearly independent vectors10 ,, nuu
Conjugate Gradients
• Gram-Schmidt: linearly independent vectors10 ,, nuu
jTj
jTi
ij
i
jjijii
dddu
dudi
ud
1
0
00
:0
Conjugate Gradients
• Gram-Schmidt: linearly independent vectors
• conjugate Gram-Schmidt:
10 ,, nuu
jTj
jTi
ij AddAdu
jTj
jTi
ij
i
jjijii
dddu
dudi
ud
1
0
00
:0
Conjugate Gradients
• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).
• CG works by setting (makes conjugate Gram-Schmidt easy)
iu
ii ru
Conjugate Gradients
• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).
• CG works by setting (makes conjugate Gram-Schmidt easy)
with1 iiii drd 11
i
Ti
iTi
i rrrr
ii ru
iu
Conjugate Gradients
• 0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
1
0 0
0:i
k jk
jTkikj
Tij
Ti rdrurdji
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
1
0 0
0:i
k jk
jTkikj
Tij
Ti rdrurdji
jiru jTi 0
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
1
0 0
0:i
k jk
jTkikj
Tij
Ti rdrurdji
jiru jTi 0
ijjTi
jTiii
rr
jirrru
0:
Conjugate Gradients
•
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
ijjTi
jTiii
rr
jirrru
0:
iTi
i
k jk
jTkiki
Tii
Ti rurdrurd
1
0 0
Conjugate Gradients
• jiAddAdr
jTj
jTi
ij
Conjugate Gradients
•
•
jiAddAdr
jTj
jTi
ij
jjjjjjjj AdrdeAAer )(11
Conjugate Gradients
•
•
jiAddAdr
jTj
jTi
ij
jjjjjjjj AdrdeAAer )(11
jTijj
Tij
Ti Adrrrrr 1
Conjugate Gradients
•
•
jiAddAdr
jTj
jTi
ij
jjjjjjjj AdrdeAAer )(11
jTijj
Tij
Ti Adrrrrr 1
1 jTij
Tij
Tij rrrrAdr
Conjugate Gradients
1 jTij
Tij
Tij rrrrAdr
10
11
jiji
jirr
jirr
Adr
rr
i
iTi
i
iTi
jTi
ijjTi
Conjugate Gradients
10 ji
ij
Conjugate Gradients
1
10
1111111
jirrrr
rdrr
Addrr
ji
iTi
iTi
iTi
iTi
def
iTii
iTiij
Method of Conjugate Gradients:
00
0
irrrdAxbr
1
)( 00max
iidrdrrrr
Axbrrrdxx
Addrr
rrrrandiiwhile
oldTold
T
old
T
T
TT
Conjugate Gradients - Convergence
Conjugate Gradients - Convergence
• A
i
Aiee 01
12
Conjugate Gradients - Convergence
•
• for steepest decent for CG Convergence of CG is much better!
A
i
Aiee 01
12