Ax=bZack

10/4/2013

Iteration method

Ax=b 𝑣1, 𝑣2… 𝑥𝑘 = 𝑉𝑘𝑦𝑘

Given (𝐴, 𝑏) standard orthonormal base 𝑣1, 𝑣2… 𝑥𝑘 = 𝑉𝑘𝑦𝑘

For symmetric A, the method to generate 𝑉𝑘 is called Lanczos Method

Lanczos

• Goal of Lanczos:

Find an orthonormal base in 𝑠𝑝𝑎𝑛(𝑏, 𝐴𝑏 …𝐴𝑘−1𝑏)

Note: this orthonormal base is unique. At k+1 iteration step, we want to generate 𝑣𝑘+1, we have:

𝛽𝑘+1𝑣𝑘+1 = 𝛾0𝑏 + 𝛾1𝐴𝑏 + 𝛾2𝐴2𝑏…𝛾𝑘−1𝐴

𝑘−1𝑏 + 𝐴𝑘𝑏𝑣𝑖𝑇𝑣𝑘+1 = 0 ∀𝑖 ≠ 𝑘

𝑣𝑘𝑇𝑣𝑘+1 = 1

So 𝑣𝑘+1 is unique.

Lanczos: Algorithm

• Given 𝑉𝑘 = 𝑣1, 𝑣2…𝑣𝑘𝛽𝑘+1𝑣𝑘+1= 𝛾0𝑏 + 𝛾1𝐴𝑏 + 𝛾2𝐴

2𝑏 …𝛾𝑘−1𝐴𝑘−1𝑏 + 𝛾𝑘𝐴

𝑘𝑏

= 𝛾′1𝑣1 + 𝛾′2𝑣2…𝛾′𝑘𝑣𝑘 + 𝛾𝑘𝐴𝑘𝑏

= 𝑧1𝑣1 + 𝑧2𝑣2…𝑧𝑘𝑣𝑘 + 𝐴𝑣𝑘= 𝐴𝑣𝑘 + 𝑉𝑘𝑧Where z is a vector, 𝛽 is a parameter to uniformization vector v

Lanczos: Algorithm

• 𝛽𝑘+1𝑣𝑘+1 = 𝐴𝑣𝑘 + 𝑉𝑘𝑧

• 𝑣𝑘𝑇𝑣𝑘+1 = 0 𝑣𝑘

𝑇𝐴𝑣𝑘 + 𝑣𝑘𝑇𝑉𝑘𝑧 = 0

𝑒𝑘𝑇𝑧 = −𝑣𝑘

𝑇𝐴𝑣𝑘

• 𝑣𝑖𝑇𝑣𝑘+1 = 0 𝑖 < 𝑘𝑣𝑖

𝑇𝐴𝑣𝑘 + 𝑣𝑖𝑇𝑉𝑘𝑧 = 0

𝑒𝑖𝑇𝑧 = 𝑣𝑖

𝑇𝐴𝑣𝑘

Lanczos: Algorithm

• Notice that we have:𝛽𝑖+1𝑣𝑖+1 = 𝐴𝑣𝑖 + 𝑉𝑖𝑧

𝑣𝑘𝑇𝛽𝑖+1𝑣𝑖+1 = 𝑣𝑘

𝑇𝐴𝑣𝑖 + 𝑣𝑘𝑇𝑉𝑖𝑧

if 𝑖 = 𝑘 − 1

𝛽𝑘 = 𝑣𝑘𝑇𝐴𝑣𝑖

else 0 = 𝑣𝑘𝑇𝐴𝑣𝑖

• 𝑒𝑖𝑇𝑧 = 𝑣𝑖

𝑇𝐴𝑣𝑘 = 𝑣𝑘𝑇𝐴𝑣𝑖 =

−𝛽𝑘 (𝑖 = 𝑘 − 1)0 (𝑖 < 𝑘 − 1)

Lanczos: Algorithm

• 𝛽𝑘+1𝑣𝑘+1 = 𝐴𝑣𝑘 − 𝑣𝑘𝑇𝐴𝑣𝑘 𝑣𝑘 − 𝛽𝑘𝑣𝑘−1

define 𝛼𝑘 = 𝑣𝑘𝑇𝐴𝑣𝑘

𝐴𝑣𝑘 = 𝛽𝑘+1𝑣𝑘+1 + 𝛼𝑘𝑣𝑘 + 𝛽𝑘𝑣𝑘−1 𝐴𝑉𝑘 = 𝑉𝑘+1𝐻𝑘

where, 𝐻𝑘 =

𝛼1 𝛽2𝛽2 𝛼2 𝛽3

⋱ ⋱ ⋱𝛽𝑘 𝛼𝑘

𝛽𝑘+1

Lanczos: Algorithm

• We can also define a tridiagonal matrix 𝑇𝑘 :

𝑇𝑘 =

𝛼1 𝛽2𝛽2 𝛼2 𝛽3

⋱ ⋱ ⋱𝛽𝑘 𝛼𝑘

;

𝐴𝑉𝑘 = 𝑉𝑘𝑇𝑘 + 𝛽𝑘+1𝑣𝑘+1𝑒𝑘𝑇

Note: theoretically, iteration will stop when 𝑘 = 𝑙 < 𝑛, but in fact, 𝑙 is always far more bigger than 𝑛, we need to set a reasonable stop rule.

• Define 𝑥𝑘 = 𝑉𝑘𝑦𝑘 is approximation of x.

• residual vector

𝑟𝑘 ≡ 𝑏 − 𝐴𝑥𝑘= 𝛽1𝑣1 − 𝐴𝑉𝑘𝑦𝑘= 𝑉𝑘+1 𝛽1𝑒1 −𝐻𝑘𝑦𝑘= 𝑉𝑘+1𝑡𝑘+1

where 𝑡𝑘+1 = 𝛽1𝑒1 −𝐻𝑘𝑦𝑘

Lanczos: Algorithm

Method Definition of optimal 𝑥𝑘

CG min 𝑟𝑘 𝐴−12

MINRES min 𝑟𝑘 22

SYMMLQ min 𝑥 − 𝑥𝑘 22

Choose “Optimal” 𝑥𝑘

• minimize function 𝑟𝑘 𝐴−12

• 𝑟𝑘 𝐴−12 = 𝑟𝑘

𝑇𝐴−1𝑟𝑘= 𝑏 − 𝐴𝑥𝑘

𝑇𝐴−1 𝑏 − 𝐴𝑥𝑘= 𝑏 − 𝐴𝑉𝑘𝑦𝑘

𝑇𝐴−1 𝑏 − 𝐴𝑉𝑘𝑦𝑘

• 𝑟𝑘 𝐴−12 has a stationary value at 𝑦𝑘 if

𝑉𝑘𝑇𝐴𝑉𝑘𝑦𝑘 = 𝑉𝑘

𝑇𝑏

𝑉𝑘𝑇(𝑉𝑘𝑇𝑘 +𝛽𝑘+1𝑣𝑘+1𝑒𝑘

𝑇)𝑦𝑘 = 𝑉𝑘𝑇𝛽1𝑣1

𝑇𝑘𝑦𝑘 = 𝛽1𝑒1

CG

• 𝑇𝑘𝑦𝑘 = 𝛽1𝑒1

• Use LDLT decomposition, 𝑇𝑘 = 𝐿𝑘𝐷𝑘𝐿𝑘𝑇 . 𝐿𝑘 is a

bidiagonal lower matrix, 𝐷𝑘 is a diagonal matrix.

• 𝐿𝑘𝐷𝑘𝐿𝑘𝑇𝑦𝑘 = 𝛽1𝑒1

• Define 𝑧𝑘 = 𝐿𝑘𝑇𝑦𝑘 , 𝐿𝑘𝐷𝑘𝑧𝑘 = 𝛽1𝑒1

• Define 𝑊𝑘𝑇 = 𝐿𝑘

−1𝑉𝑘𝑇

• 𝑥𝑘 = 𝑉𝑘𝑦𝑘 = 𝑊𝑘𝐿𝑘𝑇𝑦𝑘 = 𝑊𝑘𝑧𝑘

1 1 1

0 1

T Tk k k

k T Tk k k

L W VL

w v

1

T T T

k k k kw w v

1 1

1 1 1 1 1

2

1 1

0 0

0 101

k k T TTk k k k k k kk k k k

k k

k k k k k

L DL L D L dT L D L

dd d d

1

2

1

k k k

k k k k

d

d d

1 1 1 11 1

1 11

=0 01

k k k kk k

k k kk k k k kk k

L D L Dz zL D z e

d d d

1 1 0k k k k kd d

1

1 1 1 1

k

k k k k k k k k k k k k

k

zx W z W w W z w x w

𝐴,𝑏, 𝑣𝑘−2, 𝑣𝑘−1 𝑣𝑘 ……… (Lanczos)

𝑑𝑘−1, 𝛽𝑘λ𝑘𝑑𝑘−1, λ𝑘 , 𝛼𝑘𝑑𝑘 ………(LDL decomposition)

λ𝑘 , 𝑤𝑘−1, 𝑣𝑘𝑤𝑘 ……… (𝑊𝑘𝑇 = 𝐿𝑘

−1𝑉𝑘𝑇)

𝑑𝑘−1, 𝑑𝑘 , λ𝑘 , ζ 𝑘−1ζ

𝑘……… (𝑧𝑘 = 𝐿𝑘

𝑇𝑦𝑘)

𝑥𝑘−1, ζ 𝑘, 𝑤𝑘𝑥𝑘 ……… (𝑥𝑘 = 𝑊𝑘𝑧𝑘 )

Problem in CG method

• A should be positive definite• If A is indefinite, in practice, CG would still give out

answer but no longer be relied upon numerically.

• It used LDL decomposition • if A is an indefinite symmetric matrix, LDLT

decomposition can be tried, often success, but it does not always exist.

Method Definition of optimal 𝑥𝑘

Subproblem Factorization Estimate of x

CG min 𝑟𝑘 𝐴−12 𝑇𝑘𝑦𝑘 = 𝛽1𝑒1 𝑇𝑘 = 𝐿𝑘𝐷𝑘𝐿𝑘

𝑇 𝑥𝑘𝐶 = 𝑉𝑘𝑦𝑘

MINRES min 𝑟𝑘 22 min 𝛽1𝑒1 −𝐻𝑘𝑦𝑘 𝑄𝑘𝐻𝑘 =

𝑅𝑘

0𝑥𝑘𝑀 = 𝑉𝑘𝑦𝑘

SYMMLQ min 𝑥 − 𝑥𝑘 22 min 𝑦𝑘 s.t.

𝐻𝑘−1𝑇 𝑦𝑘 = 𝛽1𝑒1

𝐻𝑘−1𝑇 𝑄𝑘−1

𝑇

= 𝐿𝑘−1 0𝑥𝑘𝐿 = 𝑉𝑘𝑦𝑘

• CG• A must be positive definite matrix

• MINRES• Any symmetric A, 𝑟𝑘

𝑀 decreases monotonically

• Risk of cancellation error when A is indefinite

• SYMMLQ• QR factor, Any symmetric A, except Ax=b must be

consistent, Very little cancellation error, method of choice for indefinite consistent system

Unsymmetric and rectangular iteration• Set 𝛼, 𝛽 as normalized parameters of 𝑢, 𝑣 , 𝛽1𝑢1 =𝑏, 𝛼1𝑣1 = 𝐴𝑇𝑢1

follow this iteration:

generate 2 orthonormal base

𝑈𝑘 = 𝑢1 𝑢2 …𝑢𝑘 𝑉𝑘 = 𝑣1 𝑣2 …𝑣𝑘

1 1

1 1 1 1

k k k k k

T

k k k k k

u Av u

v A u v

Unsymmetric and rectangular iteration• PROOF:

assume that we have orthonormal vectors

𝑢1 𝑢2 …𝑢𝑘 , 𝑣1 𝑣2…𝑣𝑘generate 𝑢𝑘+1 , 𝑣𝑘+1 as:

1 1

1 1 1 1

k k k k k

T

k k k k k

u Av u

v A u v

Unsymmetric and rectangular iteration

1

T T T T

k i i k i k i iv v v A u v v

1 1

T T T

i k k i k i k ku u u Av u u

When i=k

1 1 1+ 0T T T T T T

k k k k k k k k k k k k k k k k ku u u Av u u v v v v u u

When i<k

1 1 1= + =0T T T T T T

i k k i k i k k k i i k i i i k ku u u Av u u v v v v u u

So 1 1k ku u u 1 1k kv v v , we can also prove that

Unsymmetric and rectangular iteration• 𝐴𝑉𝑘 = 𝑈𝑘+1𝐵𝑘 = 𝑈𝑘𝐿𝑘 + 𝛽𝑘+1𝑢𝑘+1𝑒𝑘

𝑇

• 𝐴𝑇𝑈𝑘+1 = 𝑉𝑘𝐵𝑘𝑇 + 𝛼𝑘+1𝑣𝑘+1𝑒𝑘+1

𝑇 = 𝑉𝑘+1𝐿𝑘+1𝑇

• 𝐿𝑘 =

𝛼1𝛽2 𝛼2

⋱ ⋱𝛽𝑘 𝛼𝑘

𝐵𝑘 =

𝛼1𝛽2 𝛼2

⋱ ⋱𝛽𝑘 𝛼𝑘

𝛽𝑘+1

Direct method: LUSOL

• Factor:

:

:

/j ij

T

i

T

T

l A A

u A

L L l

UU

u

A A lu

Direct method: LUSOL

• Pivot strategies

1. Preserve sparsity

Direct method: LUSOL

a Markowitz strategies is used to select potential pivot 𝐴𝑖𝑗 . Pivot should have a low Markowitz merit function

𝑀𝑖𝑗 ≡ 𝑟𝑖 − 1 𝑐𝑗 − 1

where 𝑟𝑖 and 𝑐𝑗 are number of nonzero element in row 𝑖 and column 𝑗.

The sparsest columns and rows are searched in turn: columns of length 1,rows of length 1, then columns of length 2, rows of length 2, and so on).

Direct method: LUSOL

2. Preserve stability.

check (i,j) given by Markowitz strategies with stability test:

Strategy Name Stability test

Threshold Partial Pivoting TPP 𝑙 ∞ ≤ 𝐿𝑡𝑜𝑙

Threshold Rook Pivoting TRP 𝑙 ∞𝑎𝑛𝑑 𝑢/𝐴𝑖𝑗 ∞≤ 𝐿𝑡𝑜𝑙

Threshold Complete Pivoting TCP 𝐴𝑚𝑎𝑥 ≤ 𝐿𝑡𝑜𝑙 ∗ 𝐴𝑖𝑗

Name Stability test

TPP 𝑙 ∞ ≤ 𝐿𝑡𝑜𝑙

TRP 𝑙 ∞𝑎𝑛𝑑 𝑢/𝐴𝑖𝑗 ∞≤ 𝐿𝑡𝑜𝑙

TCP 𝐴𝑚𝑎𝑥 ≤ 𝐿𝑡𝑜𝑙 ∗ 𝐴𝑖𝑗

Example, Ltol=3.0

Thank you