A matrix generation approach for eigenvalue optimization

Mohammad R. Oskoorouchi

California State University San Marcos

San Marcos, CA, USA

International Conference on Continuous Optimization

Rensselaer Polytechnic Institute

August 2 – 4, 2004

Outline:Outline:

• Eigenvalue optimization

• From optimization to feasibility

• Weighted analytic center

• Recovering feasibility

• Adding a p-dimensional matrix

• Adding a column

• Computational experience.

• Future extensions

Consider the following optimization problem:

max. .

min ( )s t

T

F y

G y h

l y u

0 1

where

( ) ,

are linearly independend

m

i ii

ni

F y F y F

F

S

Eigenvalue optimization:

max 0( ) ( )i if y F y F

Maximum eigenvalue function:

• Continuous

• Convex

• nondifferentiable

• nonpolyhedral cone

• Cannot be written as the point-wise maximum of finite number of convex smooth function

(if are linearly independent)iF

The objective function can be cast as a semidefinite program:

Now let be a feasible point and the maximum eigenvalue of has multiplicity , and be a matrix whose columns form a basis for the eigenspace for the maximum eigenvalue.

By restricting to a subcone generated at the query point, we obtain a lower bound for

( ) max{ ( ) : t ( ) 1, 0}f y F y U r U U

( ) f y

( )F yp n pQ

U

y

max

( ) max{ ( ) : t ( ) 1, 0}

( )

T

T

f y F y QUQ r U U

Q F y Q

L

b

et

e the feasible re

: , an

gi n.

d

o

m Ty G y h l y u F

where, and be upper and lower bounds o

,

n the optimal

objective

: , ( )

Define

value.

D y z z f y z

F

*

Note that

min ( )f f f y

( )f y

( ) max{ : 1,..., }Ti if y a y b i m

( ) max{ : 1,2}Ti if y a y b i

( )f y

min ( ), where is convex nondifferentiabley R

f y f

( )f y

( )f y

, ( ):mD y z zf y ¡ ¡

, : , ( )D y z z f y z F

max( ) ( )Tf y z Q F y Q z

( )TQ F y Q zI

Let us study the inequality more closely:

( )f y z

or

01

mT T

i ii

y Q FQ zI Q F Q

0T

i iQ F y F Q zI

11: , ,m T T

D my y C A y c y A

1

1

1 0

, , 1,..., ,

,

mT T

i i i ii

Tm

y y A A Q FQ i m

A I C Q F Q

A

01

mT T

i ii

y Q FQ zI Q F Q

is represented by where

T y CA

D

• is a compact convex set with nonempty interior, composed of linear and semidefinite inequalities.

• contains the optimal solution set

• is referred to as the set of localization

11: , ,m T T

D my y C A y c y A

Weighted Analytic Center

The weighted analytic is defined as the maximizer of the weighted dual potential function:

. .

1

max ( , , ) log det log log

0, 0, 0

Ds t

T

T

m

S s S s

y S C

A y s c

y

S s

f

A

The primal formulation

The weighted analytic center can be alternatively derived by the weighted primal potential function:

10, 0, 0 : 0P mX x X Ax e A

, , log det

log log

P

T

X x C X X

c x x

. .max , ,

, ,

Ps t

P

X x

X x

KKT optimality conditions

From the KKT optimality conditions 0, 0, 0

are optimal if and only if there exist 0, 0, and 0

such that

S s

X x

f

f

1

1 0

1

T

T

m

m

y S C

A y s c

y

X Ax e

XS I

xs e

A

A

Approximate analytic centers

We call , , , , , an approximate analytic center ifX x S s

1

1

2 2 2

0

1 1

T

T

m

m

y S C

A y s c

y

X Ax e

XS I xs e

A

A

The primal directions:Let a strictly primal feasible point is available.

We employ Newton method to compute the primal

directions to obtain an approximate analytic center.

2

2

x lp

dX X XSX

d x X s

d

2 21 1

2 21

where

T T TP P lp m m

P P lp m

G y g

G AX A e e

g C AX c e

A A

A

Projection to the primal null space:

Feasible directions , , should satisfyxdX d and d

2

2

T

Tx x lp

dX dX X q X

d d X A q

d d q

A

1 0x mdX Ad d e A

This condition may not satisfy due to the computational round-off error. We therefore project the primal directions back to the primal null space.

11 1x mq G dX Ad d e

A

Lemma

.5 .5 1

max

Let ( ) , ( ) , ,

where , , are primal directions computed

by the primal algorithm. Let be the eigenvalues of

and .

Then ( ) 0, ( ) 0, 0, for any 0,

i

x

x

i

i i x

X X dX x x d d

dX d and d

s

X dX X x d

X x

where

max 1,

1

min , ,i j i j d

Recovering feasibility:

So far, we reformulated the eigenvalue optimization problem into a convex feasibility problem, and derived a query point as an approximate analytic center of the set of localization:

The eigenvalue function is evaluated at this point and there are two cases:

10, 0, 0 : 0P mX x X Ax e A

11: , ,m T T

D my y C A y c y A

max 0( ) i if y F y F

Adding a p-dimensional matrix:

11: , ,

min , ( )

m TD D my y D y

f y

B

If f is not differentiable at the query point y, an oracle returns a subgradient of f at this point.

The subgradient is in the form of semidefinite inequality and will be used to update the set of localization. Dual set of localization will be updated via

01

( ) ( )m

k T k k T ki i

i

y Q FQ zI Q F Q

( )f yIllustration:

( )f yz

{ , }(: )mD y z y zzf ¡ ¡

Primal set of localization:

10, 0, 0 : 0P mX x X Ax e A

Due to computational difficulties with the deep cuts, in practice we use the primal set to recover centrality.

We now need a strictly feasible point for the updated primal set of localization.

10, 0, 0 : 0P m

Xx X T Ax e

T

A B

Primal updating direction:

1

2 221 1 1

max log det

.

0

1

0

x m

lp x

T

s t

dX T Ad d e

X dX X d d

T

A B

Implementing Newton method, one has

0arg min log det

2T

pT trT T T

V

1Twhere G V B B

Primal updating direction:

The objective function of this problem is composed of a quadratic term and a self-concordant function. Therefore, Newton method is suitable to solve this problem:

0arg min ( ) log det

2T

pT F T trT T T

V

1 1 1

( ) ( )

21

2

F T dT F T

pptr dT T tr dT dT

trT dT tr T dT T dT

V V

Primal updating direction:

The Newton step dT is obtained by setting the gradient of F(T+dT) with respect to dT to zero:

0pT T T pT dT T T dT V V

0pT T T T dT V

Adding a column:

10, 0, 0 : 0P mX x X Ax bx e A

1

2 221 1 1

max log

.

0

1

0

x m

lp x

x

s t

dX Ad bx d e

X dX X d d

x

A

Size

m

Dimension

n

Density

%

Columns Matrices CPU time

mm:ss

50

50

5

50

100

97

108

8

23

22

0

00:24

00:31

00:00

300

5

50

100

76

87

50

34

04:30

05:56

500

5

50

100

99

66

75

42

54

53

12:52

16:31

11:33

200

50

5

50

100

409

401

22

72

76

1

12:45

10:22

00:06

200

5

50

100

326

323

184

106

108

55

36:14

43:22

10:04

300

5

50

100

283

294

317

119

128

121

62:49

83:05

49:12

Size

m

Dimension

n

Density

%

Columns Matrices CPU time

mm:ss

500

10 5

100

88

23

0

0

02:28

00:39

20 5

100

203

30

0

0

06:30

00:51

50 5

100

1418

59

90

2

72:05

01:59

800

10 5

100

68

16

8

1

07:50

01:36

20 5

100

297

52

1

0

31:23

04:33

50 5

100

1311

73

99

0

438:23

05:08

1000

10 5

100

57

27

12

2

12:16

03:07

20 5

100

231

56

13

0

45:19

09:24

50 5

100

1187

105

73

0

382:10

17:41