Improving Performance of The Interior Point

Methodby Preconditioning

Project Proposal by: Ken RyalsFor: AMSC 663-664

Fall 2007-Spring 2008

2

BackgroundThe IPM method solves a sequence of constrained

optimization problems such that the sequence of solutions approaches the true solution from within the “valid” region. As the constraints are “relaxed” and the problem re-solved, the numerical properties of the problem often become more “interesting”.

μ0

3

ApplicationWhy is the IPM method of interest?

– It applies to a wide range of problem types:• Linear Constrained Optimization• Semidefinite Problems• Second Order Cone Problems

– Once in the “good region” of a solution to the set of problems in the solution path (of µ ‘s):• Convergence properties are great (“quadratic”).• It keeps the iterates in the “valid” region.

Specific Research Problem:– Optimization of Distributed Command and

Control

4

Optimization ProblemThe linear optimization problem can be formulated

follows:inf{ cTx | Ax = b}.

The search direction is implicitly defined by the system:Δx + π Δz = r

A Δx = 0AT Δy + Δz = 0.

For this, the Reduced Equation is:A π AT Δy = −Ar (= b)

From Δy we can get Δx = r − π ( −AT Δy ).

Note: The Nesterov-Todd direction corresponds to π = D⊗D, where: π z = x.i.e., D is the metric geometric mean of X and Z−1 D = X½(X½ZX½)−½X½

x is the unknowny is the “dual” of x

z is the “slack”

5

The ProblemFrom these three equations, the Reduced Equations for

Δy are:A π AT Δy = −Ar (= b)

The optimization problem is “reduced” to solving a system of linear equations to generate the next solution estimate.

If π cannot be evaluated with sufficient accuracy, solving these equations becomes pointless due to error propagation.

Aside: Numerically, evaluating ‘r − π Δz’ can also be a challenge. Namely, if the iterates converge, then Δx (= r − π Δz) approaches zero, but r does not; hence the accuracy in Δx can suffer from problems, too.

6

Example: A Poorly-Conditioned Problem

Consider a simple problem:

Let’s change A1,1 to make it ill-conditioned:11)*(

1161

)*(630364046

*100210102100112

T

TT

AACondition

AAAAA

014+3.88e 478553879241294)*(

4850091.031129460149049.03137361 000044000000000

)*(63200000023619999998-

2000000219999998-000024000000000*

1002101021001110*2 7

T

TT

AACondition

AAAAA

1 6 11 161013

1014

1015

Iteration Number

Cond

ition N

umbe

r

ADAT Condition Number

1 6 11 160

2

4

6

8

Iteration Number

Para

meter

Valu

es

Solution Parameter Paths

1 6 11 16100105101010151020102510301035

Iteration Number

Cond

ition N

umbe

r

D2 Condition Number

Constraint Matrix

7

Observations• The AD2AT condition exhibits an interesting dip

around iteration 4-5, when the solution enters the region of the answer.

• How can we exploit whatever caused the dip?– The standard approach is to use factorization to

improve numerical performance.• The Cholesky factorization is: UTU = A π AT

– Factoring a matrix into two components often trades one matrix with a condition of “M” for two matrices with conditions of ≈“√M”.

– My conjecture is that AAT and D2 interacted to lessen the impact of the ill conditioning in A

⇒Can we precondition with AAT somehow?

8

Conjecture - MathWe are solving: A π AT Δy = −Ar A is not square, so it isn’t invertible; but AAT

is…What if we pre-multiplied by it?

(AAT)-1 A π AT Δy = − (AAT)-1 ArNote: Conceptually, we have:

= Since, this looks like a similarity transform,

it might have “nice” properties…

− (AAT)-1 Ar (AAT)-1 A π AT Δy − (AAT)-1 b (AT)-1 π AT Δy

9

Conjecture - NumbersRevisit the ill-conditioned simple problem,

Condition of A π AT (π is D2) = 4.2e+014Condition of (AAT) = 4.0e+014Condition of (AAT)-1 A D2 AT = 63.053

(which is a little less than 1014)How much would it cost?

AAT is m by m (m = # constraints)neither AAT or(AAT)-1 is likely to be sparse…

Let’s try it anyway…(If it behaves nicely, it might be worth figuring out how to do it efficiently)

10

Experiment - ResultsIt does work ☺• The condition number stays low

(<1000) instead of hovering in the 1014 range.

It costs more ☹• Need inverse of AAT once.• (AAT)-1 gets used every iteration.

The inverse is needed later, rather than early in the process; thus, it could be iteratively developed during the iteration process…

1 6 11 1601234567

Iteration Number

Para

mete

r Valu

es

Solution Parameter Paths

1 6 11 16100

101

102

103

Iteration Number

Cond

ition N

umbe

r

(AAT)-1ADAT Condition Number

Solution enters region of “Newton” convergence

11

Project Proposal1. Develop a system to for preconditioned IPM.

• Create Matlab version to define structure of system.• Permits validation against SeDuMi and SPDT3

• Use C++ transition from Matlab to pure C++.• Create MEX modules from C++ code to use in Matlab

2. Apply the technique to the Distributed C2 problem3. Modify the system to develop (A*AT)-1 iteratively or to solve the

system of equations iteratively• Can we use something like the Sherman-Morrison-Woodbury Formula?

(A - ZVT )-1 = A-1 + A-1Z(I - VTA-1Z)-1VTA-1

• Can the system can be solved using the Preconditioned Conjugate Gradient Method?4. Time permitting, “parallel-ize” the system

– Inverse generation branch, and– Iteration branch using current version of Inverse.

Testing: Many test optimization problems can be found online “AFIRO” is a good start (used in AMSC 607 – Advanced Numerical Optimization)