Post on 11-Mar-2018
Sparse Linear Algebra: LU Factorization
Kristin Davies
Peter He
Feng Xie
Hamid Ghaffari
April 4, 2007
Outline
� Introduction to LU Factorization (Kristin)
� LU Transformation Algorithms (Kristin)
� LU and Sparsity (Peter)
� Simplex Method (Feng)
� LU Update (Hamid)
Introduction – Transformations – Sparsity – Simplex – Implementation
Introduction – What is LU Factorization?
� Matrix decomposition into the product of a
lower and upper triangular matrix:
LUA =
� Example for a 3x3 matrix:
=
33
2322
131211
333231
2221
11
333231
232221
131211
00
00
00
u
uu
uuu
lll
ll
l
aaa
aaa
aaa
Introduction – Transformations – Sparsity – Simplex – Implementation
Introduction – LU Existence
� LU factorization can be completed on an
invertible matrix if and only if all its principle
minors are non-zero
� Recall: invertible Recall: principle minors( ) ( ) ( )
333231
232221
131211
3331
1311
3332
2322
2221
1211
332211
det
detdetdet
detdetdet
aaa
aaa
aaa
aa
aa
aa
aa
aa
aa
aaa
=
333231
232221
131211
aaa
aaa
aaa
A
IBAAB ==
s.t.existsBifinvertibleisA
Introduction – Transformations – Sparsity – Simplex – Implementation
Introduction – LU Unique Existence
� Imposing the requirement that the diagonal of
either L or U must consist of ones results in
unique LU factorization
=
100
10
1
0
00
00
0
1
01
001
23
1312
333231
2221
11
33
2322
131211
3231
21
333231
232221
131211
u
uu
lll
ll
l
u
uu
uuu
ll
l
aaa
aaa
aaa
Introduction – Transformations – Sparsity – Simplex – Implementation
Introduction – Why LU Factorization?
� LU factorization is useful in numerical analysis
for:
– Solving systems of linear equations (AX=B)
– Computing the inverse of a matrix
� LU factorization is advantageous when there is
a need to solve a set of equations for many
different values of B
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformation Algorithms
� Modified form of Gaussian elimination
� Doolittle factorization – L has 1’s on its diagonal
� Crout factorization – U has 1’s on its diagonal
� Cholesky factorization – U=LT or L=UT
� Solution to AX=B is found as follows:
– Construct the matrices L and U (if possible)
– Solve LY=B for Y using forward substitution
– Solve UX=Y for X using back substitution
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Doolittle
� Doolittle factorization – L has 1’s on its diagonal
� General algorithm – determine rows of U from top to bottom; determine columns of L from left to right
for i=1:n
for j=i:n
LikUkj=Aij gives row i of U
end
for j=i+1:n
LjkUki=Aji gives column i of L
end
end
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Doolittle example
( ) 22
0
0001
824
364
212
00
0
1
01
001
11
11
33
2322
131211
3231
21
=⇒=
−−
−
−−
=
u
u
u
uu
uuu
ll
l
for j=i:n
LikUkj=A
ijgives row i of U
end
2,1
824
364
212
00
0
1
01
001
1312
33
2322
131211
3231
21
−=−=⇒
−−
−
−−
=
uu
u
uu
uuu
ll
l
Similarly
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Doolittle example
( ) 2424
0
0
2
01
824
364
212
00
0
212
1
01
001
212121
33
2322
3231
21
−=⇒−=⇒−=
−−
−
−−
=
−−
lll
u
uu
ll
l
for j=i+1:n
LjkUki=A
jigives column i of L
end
2
824
364
212
00
0
212
1
01
001
31
33
2322
3231
21
−=⇒
−−
−
−−
=
−−
l
u
uu
ll
l
Similarly
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Doolittle example
−−
−
−−
=
−−
824
364
212
00
0
212
1
01
001
33
2322
3231
21
u
uu
ll
l
−−
−
−−
=
824
364
212
00
0
1
01
001
33
2322
131211
3231
21
u
uu
uuu
ll
l
−−
−
−−
=
−−
−
−
824
364
212
00
0
212
12
012
001
33
2322
32 u
uu
l
−−
−
−−
=
−
−−
−
−
824
364
212
00
140
212
12
012
001
3332 ul
−−
−
−−
=
−
−−
−−
−
824
364
212
00
140
212
112
012
001
33u
−−
−
−−
=
−
−−
−−
−
824
364
212
300
140
212
112
012
001
1st row of U
2nd row of U
1st coln of L2nd coln of L
3rd row of U
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Doolittle example
� Execute algorithm for our example (n=3)
for i=1:n
for j=i:n
LikUkj=A
ijgives row i of U
end
for j=i+1:n
LjkUki=A
jigives column i of L
end
end
i=1
j=1 ���� L1kUk1=A
11gives row 1 of U
j=2 ���� L1kUk2=A
12gives row 1 of U
j=3 ���� L1kUk3=A
13gives row 1 of U
j=1+1=2 ���� L2kUk1=A
21gives column 1 of L
j=3 ���� L3kUk1=A
31gives column 1 of L
i=2
j=2 ���� L2kUk2=A
22gives row 2 of U
j=3 ���� L2kUk3=A
23gives row 2 of U
j=2+1=3 ���� L3kUk2=A
32gives column 2 of L
i=3
j=3 ���� L3kUk3=A
33gives row 3 of U
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Crout
� Crout factorization – U has 1’s on its diagonal
� General algorithm – determine columns of L from left to right; determine rows of U from top to bottom (same!?)
for i=1:n
for j=i:n
LjkUki=Aji gives column i of L
end
for j=i+1:n
LikUkj=Aij gives row i of U
end
end
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Crout example
( ) 22
0
0
1
00
824
364
212
100
10
1
0
00
1111
23
1312
333231
2221
11
=⇒=
−−
−
−−
=
ll
u
uu
lll
ll
l for j=i:n
LjkUki=A
jigives column i of L
end
Similarly
4,4
824
364
212
100
10
1
0
00
3121
23
1312
333231
2221
11
−=−=⇒
−−
−
−−
=
ll
u
uu
lll
ll
l
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Solution
� Once the L and U matrices have been found, we can easily solve our system
AX=B ���� AUX=B
=
3
2
1
3
2
1
333231
2221
11
0
00
b
b
b
y
y
y
lll
ll
l
=
3
2
1
3
2
1
33
2322
131211
00
0
y
y
y
x
x
x
u
uu
uuu
nifora
xlb
y
l
by
ii
i
j
jiji
i ,...2
1
1
11
11
=
−
=
=
∑−
=1),...1(
1 −=
−
=
=
∑+=
niforu
yuy
x
u
yx
ii
n
ij
jiji
i
nn
nn
LY=B UX=Y
Forward Substitution Backward Substitution
Introduction – Transformations – Sparsity – Simplex – Implementation
References - Intro & Transformations
� Module for Forward and Backward Substitution. http://math.fullerton.edu/mathews/n2003/BackSubstitutionMod.hml
� Forward and Backward Substitution. www.ac3.edu/papers/Khoury-thesis/node13.html
� LU Decomposition. http://en.wikipedia.org/wiki/LU_decomposition
� Crout Matrix Decomposition. http://en.wikipedia.org/wiki/Crout_matrix_decomposition
� Doolittle Decomposition of a Matrix. www.engr.colostate.edu/~thompson/hPage/CourseMat/Tutorials/C
ompMethods/doolittle.pdf
Introduction – Transformations – Sparsity – Simplex – Implementation
Definition and Storage of Sparse Matrix
� sparse … many elements are zero
for example: diag(d1,…,dn), as n>>0.
� dense … few elements are zero
� In order to reduce memory burden, introduce
some kinds of matrix storage format
Introduction – Transformations – Sparsity – Simplex – Implementation
Definition and Storage of Sparse Matrix
� Regular storage structure.
row 1 1 2 2 4 4
list = column 3 5 3 4 2 3
value 3 4 5 7 2 6
– A standard sparse matrix format
Subscripts:1 2 3 4 5 6 7 8 9 10 11
Colptr: 1 4 6 8 10 12
Rowind: 1 3 5 1 4 2 5 1 4 2 5
Value: 1 2 5 -3 4 -2 –5-1 –4 3 6
Introduction – Transformations – Sparsity – Simplex – Implementation
Definition and Storage of Sparse Matrix
1 -3 0 -1 0
0 0 -2 0 3
2 0 0 0 0
0 4 0 –4 0
5 0 –5 0 6
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (square)
�
� Matrix is of permutation and
A G(A) Adj(G)↔ ↔
iAdj(G) {0,1}
j
∈
TG(A) G(P AP)=P
T T
T T
opt{P AP P:G(P AP) G(A)}
opt{P adj(G)P P :G(P adj(G)P) G(A)}
= ⇔
=
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (square)
� 1. There is no loop in G(A)
Step 1. No loop and t = 1
Step 2. output vertices
Step 3.
∃
{ }t
t t t V(G)nt 1 2S i i . . . i ⊂=
t
t
t t tn1 2
adj(G) adj(G) \ adj(G)t t t
n1 2
i i . . . i
i i . . . i
=
t=t+1, returning to step 2.
� After p times,
(separation of set).
such that is a lower triangular
block matrix . Therefore is a lower
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (square)
{ } p1 21,2,...,n S S ... S= ∪ ∪ ∪
p1 2
TT T Ts s sP P ,P ,...,P
∃ =
TP adj(G)P
TP AP
triangular block matrix .
� 2.There is a loop in G(A).
If the graph is not strongly connected, in the
reachable matrix of adj(A), there are naught
entries.
Step 1. Choose the j-th column, t = 1,and
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (square)
∩
= = ≤ ≤ =T(R R )ti
S (j) i 1,1 i nj
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (square)
is closed on strong connection.
Step 2: Choose the -th column ( ),
, t=t+1, returning step 1.
� After p times,
{ }t
t t tn1 2
t
i i . . . i
( j S 1 2 . . . n ,
=
∈ ⊂
tS )
1j 1j j≠
1j j=
{ } p1 21,2,...,n S S ... S= ∪ ∪ ∪
is a lower triangular block matrix. Therefore
is a lower triangular block matrix.
� Note of case 2 mentioned above.
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (square)
p1 2
T T T T Ts s sP (P P . . . P ) , P adj(G)P∃ = ∋
TP AP
= =
11 12 1p
21 22 2pT
p1 pp
ˆ ˆ ˆA A . . . A
ˆ ˆ ˆA A . . . AA P adj(G)P
. . . . . .
ˆ ˆA . . . . A
.
p pij
ij ij
ij
B (b ) R ,
0 i j
ˆb 1 A 0.
ˆ0 A 0
×
= ∈
= =
= = ≠ ⇒
= =
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (square)
There is no loop of G(B). According to similar
analysis to case 1,
. Therefore, is a lower triangular block
matrix.
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (square)
p pTp p1 2 1 2
.P(j j . . . j ) BP(j j . . . j ) R
.
.
×
××
= ∈× ×× × ×
TP AP
� Reducing computation burden on solution to
large scale linear system
� Generally speaking, for large scale system,
system stability can easily be judged.
� The method is used in hierarchical optimization
of Large Scale Dynamic System
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (square)
Dulmage-Mendelsohn Decomposition
� such that
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (General)
P,Q∃
T
h
s
v
A
P AQ A X
A
x x
=
� Further, fine decomposition is needed.
the block diagonal form
the block diagonal form
the block upper triangular form.
� D-M decomposition can be seen in reference
[3].
� Computation and storage: Minimum of filling in
The detail can be see in reference [2].
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of Sparse Matrix (General)
h
v
s
A
A
A
→
→→
� left-looking. kth stage computes kth column of L and U
1. L = I
2. U = I
3. for k = 1:n
4. s = L \ A(:,k)
5. (partial pivoting on s)
6. U(1:k,k) = s(1:k)
7. L(k:n,k) = s(k:n) / U(k,k)
8. end
Introduction – Transformations – Sparsity – Simplex – Implementation
LU Factorization Method :Gilbert/Peierls
Introduction – Transformations – Sparsity – Simplex – Implementation
� THEOREM(Gilbert/Peierls). The entire
algorithm for LU factorization of A
with partial pivoting can be
implemented to run in O(flops (LU)+ m)
time on a RAM
where m is number of the nonzero
entries of A.
� Note: The theorem expresses that the LU
factorization will run in the time
within a constant factor of the best
possible, but it does not say what the
Introduction – Transformations – Sparsity – Simplex – Implementation
LU Factorization Method :Gilbert/Peierls
x = b;
for j = 1:n
if (x(j) 0) , x(j+1:n) =
x(j+1:n) - L(j+1:n,j) * x(j);
end; ---Total time: O(n+flops)
Introduction – Transformations – Sparsity – Simplex – Implementation
Sparse lower triangular solve, x=L\b
≠
Introduction – Transformations – Sparsity – Simplex – Implementation
Sparse lower triangular solve, x=L\b
�
�
� Let have an edge
if
� Let and
� Then
---Total time: O(flops)
i ib 0 x 0≠ ⇒ ≠
j ij ix 0 l 0 x 0≠ ∧ ≠ ⇒ ≠
G(L) j i→ijl 0≠
{ }ii b 0β = ≠ { }i
i x 0χ = ≠
G(L)Reach ( )χ = β
Introduction – Transformations – Sparsity – Simplex – Implementation
Sparse lower triangular solve, x=L\b
Introduction – Transformations – Sparsity – Simplex – Implementation
Sparse lower triangular solve, x=L\b
Introduction – Transformations – Sparsity – Simplex – Implementation
Sparse lower triangular solve, x=L\b
Introduction – Transformations – Sparsity – Simplex – Implementation
Sparse lower triangular solve, x=L\b
Introduction – Transformations – Sparsity – Simplex – Implementation
Sparse lower triangular solve, x=L\b
� J. R. Gilbert and T. Peierls, Sparse partial pivoting in time proportional to arithmetic operations, SIAM J.
Sci. Statist. Comput., 9 (1988), pp. 862-874
� Mihalis Yannakakis’ website:
http://www1.cs.columbia.edu/~mihalis
� A. Pothen and C. Fan, Computing the block triangular
form of a sparse matrix, ACM Trans. On Math. Soft.
Vol. 18, No.4, Dec. 1990, pp. 303-324
Introduction – Transformations – Sparsity – Simplex – Implementation
References
� Problem size determined by A
� On the average, 5~10 nonzeros per column
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Problem size
{ }min , 0,T m nc x Ax b x A R
×= ≥ ∈
greenbea.mps
of NETLIB
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Computational Form, Basis
Note: A is of full row rank and m<n
Basis (of Rm) : m linearly independent columns of A
Basic variables
{ }min , 0,T m nc x Ax b x A R
×= ≥ ∈
0 2
1 1
1 1
0 2
x1 x2 x3 x4
β index set of basic variables
γ index set of non-basic variables
B:=Aβ basis
R:=Aγ non-basis (columns)
[ | ], ,x c
A B R x cx c
β β
γ γ
= = =
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Notations
Basic feasible solution
If a problem has an optimal solution, then there is a basic
solution which is also optimal.
Ax b=
1( )x B b Rxβ γ−= −
, 0,0 x xx A bγ ≥ ==
Bx Rx bβ γ+ =
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basic Feasible Solution
Objective value :
Optimality condition :
( )1 1
0
constant Reduced Costs
T T T
T T T
T
c x c x c x
c c R x
c
B b c B
d x
β β γ γ
β γ γ
γ
β− −
= +
= −
= +
+��� �������
for all0j
d j γ≥ ∈
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Checking Optimality
non-basic variables
� Choose xq (incoming variable) s.t. dq < 0
� increase xq as much as possibleObjective value
0
Tc d xγ+
1 0
0
m
q
n
x
x
x
+
=
↑
=
⋮
⋮
basic variables
1
p
n
x
x
x
↑
→
↑
⋮
⋮
objective value is unbounded
basic variables remain feasible
(≥ 0) even if xq ���� ∞
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Improving Basic Feasible Solution
non-basic variables
� Choose xq (incoming variable) s.t. dq < 0
� increase xq as much as possible
Objective value
0
Tc d xγ+
1 0
0
m
q
n
x
x
x
+
=
↑
=
⋮
⋮
basic variables
1
p
n
x
x
x
↑
→
↑
⋮
⋮
basic variables
1
p
n
x
x
x
↑
↓
↑
⋮
⋮
goes to 0 first
(outgoing variable)
Unbounded solution neighboring improving basis
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Improving Basic Feasible Solution
1,..., ,...,p m
B b bb =
[ ]1,..., ,...,m
aB b b=
Neighboring bases :
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
1,..., ,...,p m
B b bb =
[ ]1,..., ,...,m
aB b b=1
Writem
i
i
i
v b Bva=
= =∑
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
1( )v B a
−=
(as the linear combination of the bases)
1,..., ,...,p m
B b bb =
[ ]1,..., ,...,m
aB b b=1
Writem
i
i
i
v b Bva=
= =∑
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
1( )v B a
−=
1 1 1
1( 0
1, where , , , , ,
)
,
Tp p m
p
ip
i
p p
p pi p
p
p p p
v v v vb B
v v v v
vb v
v v
v
b a
η η−
≠
+ = = −
= − ≠
− − −
∑
⋯ ⋯
1,..., ,...,p m
B b bb =
[ ]1,..., ,...,m
aB b b=1
Writem
i
i
i
v b Bva=
= =∑
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
1( )v B a
−=
1 1 1
1
1, ,where , , , , ,
p
i
ip pi p
Tp p
p
m
p p p p p
vb
v v
v v v vB
v v v v v
b a
b η η
≠
− +
= −
= = − − − −
∑
⋯ ⋯
vp : pivot element
1,..., ,...,p m
B b bb =
[ ]1,..., ,...,m
aB b b=1
Writem
i
i
i
v b Bva=
= =∑
1 1 1,..., , , ,., where ..,p p m
B BE E e e e eη− + = =
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
(elementary transformation matrix)
1( )v B a
−=
1 1 1
1
1, ,where , , , , ,
p
i
ip pi p
Tp p
p
m
p p p p p
vb
v v
v v v vB
v v v v v
b a
b η η
≠
− +
= −
= = − − − −
∑
⋯ ⋯
1,..., ,...,p m
B b bb =
[ ]1,..., ,...,m
aB b b=1
Writem
i
i
i
v b Bva=
= =∑
1 1 1,..., , , ,., where ..,p p m
B BE E e e e eη− + = =
1 1B EB
− −=
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
1( )v B a
−=
1 1 1
1
1, ,where , , , , ,
p
i
ip pi p
Tp p
p
m
p p p p p
vb
v v
v v v vB
v v v v v
b a
b η η
≠
− +
= −
= = − − − −
∑
⋯ ⋯
1 1 1
1
1
, ..., , , , ...,
1
1
1 0 1 01
11
0 1 10 1
p p m
p
m
p
m
E e e e eη
η
η
η
η
η
η
− + =
=
=
⋱
⋱
⋱ ⋮ ⋱ ⋮⋱ ⋮
⋯
⋮ ⋱ ⋮ ⋱⋮ ⋱
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
ETM
(Elementary Transformation Matrix)
Basis tends to get denser after each update
1 1 1 11
1
p
p p p p
m m m p m
w w w
Ew w w
w w w
η η
η η
η η
+ = = +
⋱ ⋮ ⋮
⋱ ⋮ ⋮
1 1( )B EB− −=
Ew = w if wp = 0
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
Steps Major ops
1. Find an initial feasible basis B
2. Initialization
3. Check optimality
4. Choose incoming variable xq
5. Choose outgoing variable xp
6. Update basis
1B b
−
1Tc Bβ−
1
qB a−
1EB
−
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Algorithm
Steps Major ops
1. Find an initial feasible basis B
2. Initialization
3. Check optimality
4. Choose incoming variable xq
5. Choose outgoing variable xp (pivot step)
6. Update basis
1B b
−
1Tc Bβ−
1
qB a−
1EB
−
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Algorithm
Choice of pivot (numerical considerations)
� resulting less fill-ins
� large pivot element
Conflicting goals sometimes
In practice, compromise.
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Algorithm
Typical operations :
Challenge: sparsity of B-1 could be destroyed by basis update
Need a proper way to represent B-1
Two ways:
� Product form of the inverse (obsolete)
� LU factorization
1 1, TB w w B
− −
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Typical Operations in Simplex Method
1
1 1( )k kB E E E−−= ⋯
� Reduce complexity using LU update
Side effect : more LU factors
� Refactorization
(reinstate efficiency and numerical accuracy)
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– LU Factorization
1 1( , )B BE B EB− −= =
Sparse LU Updates in Simplex Method
Hamid R. Ghaffari
April 10, 2007
Outline
LU Update MethodsPreliminariesBartels-Golub LU UPDATESparse Bartels-Golub MethodReid’s MethodThe Forrest-Tomlin MethodSuhl-Suhl MethodMore Details on the Topic
Revised Simplex Algorithm
Simplex Method Revised Simplex Method
Determine the current basis, d d = B−1b
Choose xq to enter the basis based onthe greatest cost contribution
c = c ′N − c ′BB−1N,{q|cq = min
t(ct)}
If xq cannot decrease the cost,d is optimal solution
cq ≥ 0, d is optimal solution
Determine xp that leaves the basis(become zero) as xq increases.
w = B−1Aq,{p di
wi= min
t
(dtwt
), wt > 0
}If xq can increase without causinganother variable to leave the basis,the solution is unbounded
If wp ≤ 0 for all i , the solution isunbounded.
Update dictionary. Update B−1
Note: In general we do not compute the inverse.
Revised Simplex Algorithm
Simplex Method Revised Simplex Method
Determine the current basis, d d = B−1b
Choose xq to enter the basis based onthe greatest cost contribution
c = c ′N − c ′BB−1N,{q|cq = min
t(ct)}
If xq cannot decrease the cost,d is optimal solution
cq ≥ 0, d is optimal solution
Determine xp that leaves the basis(become zero) as xq increases.
w = B−1Aq,{p di
wi= min
t
(dtwt
), wt > 0
}If xq can increase without causinganother variable to leave the basis,the solution is unbounded
If wp ≤ 0 for all i , the solution isunbounded.
Update dictionary. Update B−1
Note: In general we do not compute the inverse.
Revised Simplex Algorithm
Simplex Method Revised Simplex Method
Determine the current basis, d d = B−1b
Choose xq to enter the basis based onthe greatest cost contribution
c = c ′N − c ′BB−1N,{q|cq = min
t(ct)}
If xq cannot decrease the cost,d is optimal solution
cq ≥ 0, d is optimal solution
Determine xp that leaves the basis(become zero) as xq increases.
w = B−1Aq,{p di
wi= min
t
(dtwt
), wt > 0
}If xq can increase without causinganother variable to leave the basis,the solution is unbounded
If wp ≤ 0 for all i , the solution isunbounded.
Update dictionary. Update B−1
Note: In general we do not compute the inverse.
Problems with Revised Simplex Algorithm
I The physical limitations of a computer can become a factor.
I Round-off error and significant digit loss are common problems inmatrix manipulations (ill-conditioned matrices).
I It also becomes a task in numerical stability.
I It take m2(m− 1) multiplications and m(m− 1) additions, a total ofm3 −m floating-point (real number) calculations.
Many variants of the Revised Simplex Method have been designed toreduce this O(m3)-time algorithm as well as improve its accuracy.
Introducing Spike
I If Aq is the entering column, B the original basis and B the newbasis, then we have
B = B + (Aq − Bep)eTq ,
I Having LU decomposition B = LU we have
L−1B = U + (L−lAq − Uep)eTq ,
II How to deal with this?
The various implementations and variations of the Bartels-Golubgenerally diverge with the next step: reduction of the spiked uppertriangular matrix back to an upper-triangular matrix. (Chvatal, p150)
Introducing Spike
I If Aq is the entering column, B the original basis and B the newbasis, then we have
B = B + (Aq − Bep)eTq ,
I Having LU decomposition B = LU we have
L−1B = U + (L−lAq − Uep)eTq ,
II How to deal with this?
The various implementations and variations of the Bartels-Golubgenerally diverge with the next step: reduction of the spiked uppertriangular matrix back to an upper-triangular matrix. (Chvatal, p150)
Introducing Spike
I If Aq is the entering column, B the original basis and B the newbasis, then we have
B = B + (Aq − Bep)eTq ,
I Having LU decomposition B = LU we have
L−1B = U + (L−lAq − Uep)eTq ,
II How to deal with this?
The various implementations and variations of the Bartels-Golubgenerally diverge with the next step: reduction of the spiked uppertriangular matrix back to an upper-triangular matrix. (Chvatal, p150)
Introducing Spike
I If Aq is the entering column, B the original basis and B the newbasis, then we have
B = B + (Aq − Bep)eTq ,
I Having LU decomposition B = LU we have
L−1B = U + (L−lAq − Uep)eTq ,
II How to deal with this?
The various implementations and variations of the Bartels-Golubgenerally diverge with the next step: reduction of the spiked uppertriangular matrix back to an upper-triangular matrix. (Chvatal, p150)
Introducing Spike
I If Aq is the entering column, B the original basis and B the newbasis, then we have
B = B + (Aq − Bep)eTq ,
I Having LU decomposition B = LU we have
L−1B = U + (L−lAq − Uep)eTq ,
II How to deal with this?
The various implementations and variations of the Bartels-Golubgenerally diverge with the next step: reduction of the spiked uppertriangular matrix back to an upper-triangular matrix. (Chvatal, p150)
Bartels-Golub MethodIllustration
The first variant of the Revised Simplex Method was the Bartels-GolubMethod.
Bartels-Golub MethodAlgorithm
Revised Simplex Method Bartels-Golub
d = B−1b d = U−1L−1b
c = c ′N − c ′BB−1N,{q|cq = min
t(ct)}
c = c ′N − c ′BU−1L−1N,{q|cq = min
t(ct)}
cq ≥ 0, d is optimal solution cq ≥ 0, d is optimal solution
w = B−1Aq,{p di
wi= min
t
(dtwt
), wt > 0
} w = U−1L−1Aq,{p di
wi= min
t
(dtwt
), wt > 0
}If wp ≤ 0 for all i , the solution isunbounded.
If wp ≤ 0 for all i , the solution isunbounded.
Update B−1 Update U−1 and L−1
Bartels-Golub MethodCharacteristics
I It significantly improved numerical accuracy.
I Can we do better?
In sparse case, yes.
Bartels-Golub MethodCharacteristics
I It significantly improved numerical accuracy.
I Can we do better? In sparse case, yes.
Sparse Bartels-Golub Methodeta matrices
First take a look at the following facts:
Column-Eta factorization of triangular matrices:
1l21 1l31 l32 1l41 l42 l43 1
=
11
1l43 1
·
11l32 1l42 1
·
1l21 1l31 1l41 1
Single-Entry-Eta Decomposition:
1l21 1l31 1l41 1
=
1l21 1
11
·
11
l31 11
·
11
1l41 1
So L can be expressed as the multiplication of single-entry eta matrices,and hence, L−1 is also is the product of the same matrices withoff-diagonal entries negated.
Sparse Bartels-Golub Methodeta matrices
First take a look at the following facts:
Column-Eta factorization of triangular matrices:
1l21 1l31 l32 1l41 l42 l43 1
=
11
1l43 1
·
11l32 1l42 1
·
1l21 1l31 1l41 1
Single-Entry-Eta Decomposition:
1l21 1l31 1l41 1
=
1l21 1
11
·
11
l31 11
·
11
1l41 1
So L can be expressed as the multiplication of single-entry eta matrices,and hence, L−1 is also is the product of the same matrices withoff-diagonal entries negated.
Sparse Bartels-Golub Methodeta matrices
First take a look at the following facts:
Column-Eta factorization of triangular matrices:
1l21 1l31 l32 1l41 l42 l43 1
=
11
1l43 1
·
11l32 1l42 1
·
1l21 1l31 1l41 1
Single-Entry-Eta Decomposition:
1l21 1l31 1l41 1
=
1l21 1
11
·
11
l31 11
·
11
1l41 1
So L can be expressed as the multiplication of single-entry eta matrices,and hence, L−1 is also is the product of the same matrices withoff-diagonal entries negated.
Sparse Bartels-Golub MethodAlgorithm
Bartels-Golub Method Sparse Bartels-Golub Method
d = U−1L−1b d = U−1 ∏t ηtb
c = c ′N − c ′BU−1L−1N,{q|cq = min
t(ct)}
c = c ′N − c ′BU−1 ∏t ηtN,
{q|cq = mint
(ct)}cq ≥ 0, d is optimal solution cq ≥ 0, d is optimal solution
w = U−1L−1Aq,{p di
wi= min
t
(dtwt
), wt > 0
} w = U−1 ∏t ηtAq,{
p diwi
= mint
(dtwt
), wt > 0
}If wp ≤ 0 for all i , the solution isunbounded.
If wp ≤ 0 for all i , the solution isunbounded.
Update U−1 and L−1 Update U−1 and create anynecessary eta matrices.If there are too many eta matrices,completely refactor the basis.
Sparse Bartels-Golub MethodAlgorithm
Bartels-Golub Method Sparse Bartels-Golub Method
d = U−1L−1b d = U−1 ∏t ηtb
c = c ′N − c ′BU−1L−1N,{q|cq = min
t(ct)}
c = c ′N − c ′BU−1 ∏t ηtN,
{q|cq = mint
(ct)}cq ≥ 0, d is optimal solution cq ≥ 0, d is optimal solution
w = U−1L−1Aq,{p di
wi= min
t
(dtwt
), wt > 0
} w = U−1 ∏t ηtAq,{
p diwi
= mint
(dtwt
), wt > 0
}If wp ≤ 0 for all i , the solution isunbounded.
If wp ≤ 0 for all i , the solution isunbounded.
Update U−1 and L−1 Update U−1 and create anynecessary eta matrices.If there are too many eta matrices,completely refactor the basis.
Sparse Bartels-Golub MethodAlgorithm
Bartels-Golub Method Sparse Bartels-Golub Method
d = U−1L−1b d = U−1 ∏t ηtb
c = c ′N − c ′BU−1L−1N,{q|cq = min
t(ct)}
c = c ′N − c ′BU−1 ∏t ηtN,
{q|cq = mint
(ct)}cq ≥ 0, d is optimal solution cq ≥ 0, d is optimal solution
w = U−1L−1Aq,{p di
wi= min
t
(dtwt
), wt > 0
} w = U−1 ∏t ηtAq,{
p diwi
= mint
(dtwt
), wt > 0
}If wp ≤ 0 for all i , the solution isunbounded.
If wp ≤ 0 for all i , the solution isunbounded.
Update U−1 and L−1 Update U−1 and create anynecessary eta matrices.If there are too many eta matrices,completely refactor the basis.
Sparse Bartels-Golub MethodAlgorithm
Bartels-Golub Method Sparse Bartels-Golub Method
d = U−1L−1b d = U−1 ∏t ηtb
c = c ′N − c ′BU−1L−1N,{q|cq = min
t(ct)}
c = c ′N − c ′BU−1 ∏t ηtN,
{q|cq = mint
(ct)}cq ≥ 0, d is optimal solution cq ≥ 0, d is optimal solution
w = U−1L−1Aq,{p di
wi= min
t
(dtwt
), wt > 0
} w = U−1 ∏t ηtAq,{
p diwi
= mint
(dtwt
), wt > 0
}If wp ≤ 0 for all i , the solution isunbounded.
If wp ≤ 0 for all i , the solution isunbounded.
Update U−1 and L−1 Update U−1 and create anynecessary eta matrices.If there are too many eta matrices,completely refactor the basis.
Sparse Bartels-Golub Method Advantages
I It is no more complex than the Bartels-Golub Method.
I Instead of just L and U, the factors become the lower-triangular etamatrices and U.
I The eta matrices were reduced to single-entry eta matrices.
I Instead of having to store the entire matrix, it is only necessary tostore the location and value of the off-diagonal element for eachmatrix.
I Refactorizations occur less than once every m times, so thecomplexity improves significantly to O(m2).
Sparse Bartels-Golub Method Advantages
I It is no more complex than the Bartels-Golub Method.
I Instead of just L and U, the factors become the lower-triangular etamatrices and U.
I The eta matrices were reduced to single-entry eta matrices.
I Instead of having to store the entire matrix, it is only necessary tostore the location and value of the off-diagonal element for eachmatrix.
I Refactorizations occur less than once every m times, so thecomplexity improves significantly to O(m2).
Sparse Bartels-Golub Method Advantages
I It is no more complex than the Bartels-Golub Method.
I Instead of just L and U, the factors become the lower-triangular etamatrices and U.
I The eta matrices were reduced to single-entry eta matrices.
I Instead of having to store the entire matrix, it is only necessary tostore the location and value of the off-diagonal element for eachmatrix.
I Refactorizations occur less than once every m times, so thecomplexity improves significantly to O(m2).
Sparse Bartels-Golub Method Advantages
I It is no more complex than the Bartels-Golub Method.
I Instead of just L and U, the factors become the lower-triangular etamatrices and U.
I The eta matrices were reduced to single-entry eta matrices.
I Instead of having to store the entire matrix, it is only necessary tostore the location and value of the off-diagonal element for eachmatrix.
I Refactorizations occur less than once every m times, so thecomplexity improves significantly to O(m2).
Sparse Bartels-Golub Method Advantages
I It is no more complex than the Bartels-Golub Method.
I Instead of just L and U, the factors become the lower-triangular etamatrices and U.
I The eta matrices were reduced to single-entry eta matrices.
I Instead of having to store the entire matrix, it is only necessary tostore the location and value of the off-diagonal element for eachmatrix.
I Refactorizations occur less than once every m times, so thecomplexity improves significantly to O(m2).
Sparse Bartels-Golub Method Disadvantages
I Eventually, the number of eta matrices will become so large that itbecomes cheaper to decompose the basis.
I Such a refactorization may occur prematurely in an attempt topromote stability if noticeable round-off errors begin to occur.
I In practice, in solving large sparse problem, the basis is refactorizedquite frequently, often after every twenty iterations of so. (Chvatal,p. 111)
I If the spike always occurs in the first column and extends to thebottom row, the Sparse Bartels-Golub Method becomes worse thanthe Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposedresulting in huge amounts of fill-in;
I Large numbers of eta matrices;I O(n3)-cost decomposition;
Sparse Bartels-Golub Method Disadvantages
I Eventually, the number of eta matrices will become so large that itbecomes cheaper to decompose the basis.
I Such a refactorization may occur prematurely in an attempt topromote stability if noticeable round-off errors begin to occur.
I In practice, in solving large sparse problem, the basis is refactorizedquite frequently, often after every twenty iterations of so. (Chvatal,p. 111)
I If the spike always occurs in the first column and extends to thebottom row, the Sparse Bartels-Golub Method becomes worse thanthe Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposedresulting in huge amounts of fill-in;
I Large numbers of eta matrices;I O(n3)-cost decomposition;
Sparse Bartels-Golub Method Disadvantages
I Eventually, the number of eta matrices will become so large that itbecomes cheaper to decompose the basis.
I Such a refactorization may occur prematurely in an attempt topromote stability if noticeable round-off errors begin to occur.
I In practice, in solving large sparse problem, the basis is refactorizedquite frequently, often after every twenty iterations of so. (Chvatal,p. 111)
I If the spike always occurs in the first column and extends to thebottom row, the Sparse Bartels-Golub Method becomes worse thanthe Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposedresulting in huge amounts of fill-in;
I Large numbers of eta matrices;I O(n3)-cost decomposition;
Sparse Bartels-Golub Method Disadvantages
I Eventually, the number of eta matrices will become so large that itbecomes cheaper to decompose the basis.
I Such a refactorization may occur prematurely in an attempt topromote stability if noticeable round-off errors begin to occur.
I In practice, in solving large sparse problem, the basis is refactorizedquite frequently, often after every twenty iterations of so. (Chvatal,p. 111)
I If the spike always occurs in the first column and extends to thebottom row, the Sparse Bartels-Golub Method becomes worse thanthe Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposedresulting in huge amounts of fill-in;
I Large numbers of eta matrices;I O(n3)-cost decomposition;
Sparse Bartels-Golub Method Disadvantages
I Eventually, the number of eta matrices will become so large that itbecomes cheaper to decompose the basis.
I Such a refactorization may occur prematurely in an attempt topromote stability if noticeable round-off errors begin to occur.
I In practice, in solving large sparse problem, the basis is refactorizedquite frequently, often after every twenty iterations of so. (Chvatal,p. 111)
I If the spike always occurs in the first column and extends to thebottom row, the Sparse Bartels-Golub Method becomes worse thanthe Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposedresulting in huge amounts of fill-in;
I Large numbers of eta matrices;I O(n3)-cost decomposition;
Sparse Bartels-Golub Method Disadvantages
I Eventually, the number of eta matrices will become so large that itbecomes cheaper to decompose the basis.
I Such a refactorization may occur prematurely in an attempt topromote stability if noticeable round-off errors begin to occur.
I In practice, in solving large sparse problem, the basis is refactorizedquite frequently, often after every twenty iterations of so. (Chvatal,p. 111)
I If the spike always occurs in the first column and extends to thebottom row, the Sparse Bartels-Golub Method becomes worse thanthe Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposedresulting in huge amounts of fill-in;
I Large numbers of eta matrices;
I O(n3)-cost decomposition;
Sparse Bartels-Golub Method Disadvantages
I Eventually, the number of eta matrices will become so large that itbecomes cheaper to decompose the basis.
I Such a refactorization may occur prematurely in an attempt topromote stability if noticeable round-off errors begin to occur.
I In practice, in solving large sparse problem, the basis is refactorizedquite frequently, often after every twenty iterations of so. (Chvatal,p. 111)
I If the spike always occurs in the first column and extends to thebottom row, the Sparse Bartels-Golub Method becomes worse thanthe Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposedresulting in huge amounts of fill-in;
I Large numbers of eta matrices;I O(n3)-cost decomposition;
Ried Suggestion on Sparse Bartels-Golub Method
Rather than completely refactoring the basis, applying LU-decompositiononly to the part of that remained upper-Hessenberg.
Reid’s Method
Task: The task is to find a way to reduce that bump before attempting todecompose it;
Row singleton: any row of the bump that only has one non-zeroentry.
Column singleton: any column of the bump that only has onenon-zero entry.
Method:I When a column singleton is found, in a bump, it is moved to the top
left corner of the bump.
I When a row singleton is found, in a bump, it is moved to the bottomright corner of the bump.
Reid’s MethodColumn Rotation
Reid’s MethodRow Rotation
Reid’s MethodCharacteristics
Advantages:
I It significantly reduces the growth of the number of eta matrices inthe Sparse Bartels-Golub Method
I So, the basis should not need to be decomposed nearly as often.
I The use of LU-decomposition on any remaining bump still allowssome attempt to maintain stability.
Disadvantages:
I The rotations make absolutely no allowance for stability whatsoever,
I So, Reid’s Method remains numerically less stable than the SparseBartels-Golub Method.
The Forrest-Tomlin Method
The Forrest-Tomlin Method
Bartels-Golub Method Forrest-Tomlin Methodd = U−1L−1b d = U−1
∏t RtL
−1bc = c ′N − c ′BU−1L−1N,{q|cq = min
t(ct)}
c = c ′N − c ′BU−1∏
t RtL−1N,
{q|cq = mint
(ct)}cq ≥ 0, d is optimal solution cq ≥ 0, d is optimal solutionw = U−1L−1Aq,{
p di
wi= min
t
(dt
wt
), wt > 0
} w = U−1∏
t RtL−1Aq,{
p di
wi= min
t
(dt
wt
), wt > 0
}If wp ≤ 0 for all i , the solution isunbounded.
If wp ≤ 0 for all i , the solution isunbounded.
Update U−1 and L−1 Update U−1 creating a row factoras necessary. If there are toomany factors, completely refactorthe basis.
The Forrest-Tomlin Method CharacteristicsAdvantages:
I At most one row-eta matrix factor will occur for each iterationwhere an unpredictable number occurred before.
I The code can take advantage of such knowledge for predictingnecessary storage space and calculations.
I Fill-in should also be relatively slow, since fill-in can only occurwithin the spiked column.
Disadvantages:
I Sparse Bartels-Golub Method allowed LU-decomposition to pivot fornumerical stability, but Forrest-Tomlin Method makes no suchallowances.
I Therefore, severe calculation errors due to near-singular matrices aremore likely to occur.
Suhl-Suhl Method
This method is a modification of Forrest-Tomlin Method.
For More Detail
Leena M. Suhl, Uwe H. SuhlA fast LU update for linear programming.Annals of Operations Research 43(1993)33-47
Stiven S. MorganA Comparison of Simplex Method AlgorithmsUniversity of Florida, 1997
Vasek ChvatalLinear ProgrammingW.H. Freeman & Company (September 1983)
Thanks